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1. INTRODUCTION 


The Large Area' Crop Inventory Experiment (LAC IE) is an inter- 
agency endeavor of the National Aeronautics and Space Administra- 
tion (NASA) , the National Oceanic and Atmospheric Administration 
(NCAA), and the U.S. Department of Agriculture (USDA) . Its pur- 
poses are to test and demonstrate the accuracy and economic 
importance of utilizing remotely sensed data from the Land 
Satellite (Landsat) in conjunction with climatological and 
conventional agriculture data to produce timely estimates of the 
production of a major world crop prior to harvest. 

The LACIE Accuracy Assessment (AA) effort is designed to check 
the accuracy of LACIE products (estimates of wheat production, 
area, and yield) throughout the growing season, to determine if 
the operational procedures are sufficient to satisfy LACIE 
project goals and objectives, and to identify problem areas in 
the estimation process. 

Most of the AA studies are conducted in the U.s. Great Plains 
(USGP) region and Canada since these regions have the most 
ground- truth data available. However, the AA studies in these 
regions are also designed to promote the development of LACIE 
procedures which can be used to obtain accurate estimates for 
other parts of the world. 

1.1 PHASE III AA OBJECTIVES 

The following are the specific objectives for AA in Phase III. 

a. To make comparisons throughout the growing season between 
the LACIE and the USDA Statistical Reporting Service (SRS) 
reference standard estimates of wheat production, area, 
and yield for the USGP region. 
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b. To determine whether the LAC IB production estimates are 
meeting the 90-90 project goal (to be within 10 percent of 
'the true value with a confidence of 90 percent) . 

c. To conduct investigations of the error sources in the LACIE 
estimates and, where possible, to relate these error sources 
to causal elements in the LACIE estimation processes. 

d. To assess the accuracy of the LACIE estimates for the U.S.S.R. 
to the extent permitted by the available data. This will 
include comparisons throughout the growing season between the 
LACIE estimates and the USDA Foreign Agricultural Service (FAS) 
estimates for the U.S.S.R. 

e. To investigate the accuracy of LACIE proportion estimates in 
the U.S. Great Plains and in selected regions of Canada, 
using blind site data. 

This document was prepared by Lockheed Electronics Company, Inc. 
(LEC) , Systems and Services Division, Houston, Texas, under con- 
tract NAS 9-15200 for the Earth Observations Division, Space and 
Life Sciences Directorate, Lyndon B. Johnson Space Center (JSC) 
of NASA. Inputs were received from NASA, USDA, and NOAA 
personnel. 
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2. BACKGROUND 


In tfie LACIE project, remote sensing technology is used in con- 
junction with meteorological and conventional agricultural infor- 
mation to examine three global crop seasons, each of which is 
designated as a LACIE phase. 

Phase I, which. began in January 1975, was devoted primarily to 
identifying, and estimating wheat acreage in the USGP states of 
Colorado,. Kansas, Minnesota, Montana, Nebraska, North Dakota, 
Oklahoma, South Dakota, and Texas. Classification analyses were 
conducted in other selected areas, and yield model development 
and yield feasibility determinations were made over selected 
regions in the United States. Data from the USDA/SRS. were com- 
pared with LACIE estimates to ’.determine the accuracy of LACIE 
performance. 

Phase I AA activities were initiated in July 1975, and tests for 
the accuracy of wheat acreage estimates were conducted using seg-. 
merits for which ground-truth data were available. Initially, 
statistical tests and comparisons of LACIE estimates with ground- 
truth data were made using data from 27 intensive test sites 
(ITS's) in eight states; then, to test a greater number of 
acquisitions in a more concentrated area, ground observations 
of harvested small grains were obtained from 30 LACIE operational 
segments in North Dakota and Montana. The identity of these 
sites was withheld from the Classification and Mensuration Sub- 
system (CAMS) analysts so that they would process them as ordi- 
nary segments. For this reason, they. were called blind sites. 
After the data from the blind sites were processed by CAMS., the 
AA Team (appendix A) compared the results of the various sampling 
and classification procedures used. Approximately 340 special 
classification runs were conducted to support Phase I AA 
activities . 
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In Phase II, which began in October 1975 and covered the 1975-76 
growing season, emphasis was on the evaluation of LACIE proce- 
dures over the USGP region and on the development of AA 
methodology. 

. ) 

LACIE acreage, yield, and production estimates for spring and 
winter wheat were compared with the corresponding USDA/SRS 
estimates. This was done for the USGP region, for various sub- 
regions of the USGP (see section 4.2), and for each state in the 
USGP. Detailed error source investigations were made by comparing 
LACIE proportion estimates with ground-truth proportions for 
over 150 blind sites and 27 ITS's. .Estimates of the production, 
coefficient of variation (CV) , and bias were used to evaluate 
the 90-90 criterion at the USGP level; and a sensitivity analysis 
was performed to determine the effect of various errors on the 
LACIE production estimate. In the foreign area, 10 ITS’s in 
Canada were studied and evaluated. 

In Phase III, which began in October 1976, the emphasis of AA 
continues to be the detailed evaluation of LACIE estimates and 
procedures over the nine-state USGP "yardstick" wheat region. 
Investigations will also be carried out in the U..S.S.R. and 
Canada, as mentioned in section 1.1. 


2-2 



3 . SCOPE 


To accomplish its objectives, AA requires the multiagency collec- 
tion of various types of data to support the Phase III evalua- 
tions. This includes: 

a. Aircraft photography over all U.S. blind sites and ITS ' s and 
Canadian test sites and ITS ' s in order to prepare maps for 
land-use annotations by USDA Agricultural Stabilization and 
Conservation Service (ASCS) personnel. 

b. Land-use annotations from ground observations by USDA/ASCS 
personnel over all U.S. blind sites and ITS ' s and Canadian 
test sites and ITS's. 

c. Special ground observances of plant height and ground cover 
data every 18 days over 15 fields from each U.S. blind site. 

d. LACIE imagery, interpretation, and classification data over 
all U.S. blind sites and ITS's and Canadian test sites end 
ITS’s. 

e. Monthly wheat production, area, and yield estimates over the 
USGP from LACIE and USDA/SRS. 

f. Monthly vjheat production, area, and yield estimates over 
regions of the U.S.S.R. and Canada from LACIE and USDA/FAS . 

The following is the distribution of blind sites, test sices, 

and ITS’s in the United States and Canada. 

a. 143 winter wheat blind sites in the USGP 

b. 68 spring wheat blind sites in the USGP 

c. 30 spring wheat test sites in Saskatchewan (Canada) 

d. 24 ITS's in the USGP 

e. 10 ITS's in the Canadian spring wheat region 


-Mq page bunk not 
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Specific details of the scope and data requirements for Phase III 
AA are presented in table 3-i. 


Listings and locations of 
by state are presented in 


ITS's and distributions of blind sites 
appendix B. 
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TABLE 3-l. — SUMMARY OF PHASE III ACCURACY ASSESSMENT DATA REQUIREMENTS 
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4. GENERAL TECHNICAL APPROACH 


Three groups of activities are required to implement the LACIE. 
Phase III AA plan and satisfy its objectives: 

a. Planning and data acquisition 

b. Data analysis and evaluation 

c . Reporting 

A detailed tlow diagram of the Phase III AA program and related 
LAC IE operational activities is presented in figure 4-1. 

Specific descriptions of the AA tasks associated with each group 
of activities are presented in section 6. 

4 . 1 PLANNING AND DATA ACQUISITION 

The first group of Phase III AA activities consists of identi- 
fying AA data requirements and monitoring the acquisition of the 
data. Most of the data will be acquired by LACIE operations - 
(NASA) , the USDA, and NOAA; but coordination is required by AA 
personnel to ensure that the data collected are adequate to 
support the AA program. 

Part of this monitoring activity is conducted by members of the 
AA Team (appendix A) . In particular, they select the blind sites 
in the United States and Canada and coordinate action for 
acquiring ground-truth data from the blind sites and ITS's in the 
United States and Canada. 


The specific tasks to implement the above activity are identified 
as tasks 1 through 8 in figure 4-1. Specific descriptions of 
these tasks are provided in section 6.1. 
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4.2 DATA ANALYSIS AND EVALUATION 


The second group of Phase III AA activities involves the analysis 
of the data collected during the Phase III data acquisition 
activities. Various subsets of the Phase III data are prepared 
to support the analyses performed by AA. These basic data sets 
are identified by the numbers in the square symbols in figure 4-1. 
Similar numbers on the right side of the diagram indicate the 
specific analysis to which each data set is' applied,' 

The Phase III AA evaluations are carried out at the country level 
for the U.S.S.R., at -selected sites in the Province of Saskatche- 
wan in Canada, and- in the nine-state USGP "yardstick" region in 
the United States. Within this region, evaluations are usually 
made for each individual state and for the following smaller 
regions. 

a. The U.S. southern Great Plains region (USSGP) — This region 
consists of Colorado, Kansas, Nebraska, Oklahoma, and Texas. 
These 3tates have winter wheat only and therefore could- also 
be called the "winter wheat states." LACIE estimates of 
wheat production are available for the USSGP from February 
through October. 

b. The spring wheat states [ (SW states), Minnesota and North 
Dakota] — These states have spring wheat only. LACIE esti- 
mates of wheat production are available from August through 
October. 

c. The mixed wheat states [(MW states)-, Montana and South 
Dakota] — These states have both spring and winter wheat. 

LACIE estimates of wheat production are available from 
August through October for spring wheat and from February 
through October for winter wheat. 

d. The U.S. northern Great Plains region (USNGP) — This region 
consists of the two spring wheat states and the two mixed 
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e. The USGP region - This region consists of the nine states of 
the USSGP and the USNGP. 

To determine the .magnitude and components of error in LACIE esti- 
mates and to ascertain whether or not the LACIE production esti- 
mates are satisfying the 90-90 criterion, AA personnel do the 
following: 

a. Determine the relative differences^ between LACIE and U3DA/ 
SRS estimates of wheat production, area, and yield over the - 
various regions of the USGP "yardstick" region and between 
LACIE and USDA/FAS estimates of wheat production, area, and 
yield in the U.S.S.R. In addition, significance tests are 
made to compare these estimates. These data are- reported 
throughout the Phase III growing season in the various AA 
quick-look reports, in the interim AA reports, and in the 
final AA report. 

b. . Determine if LACIE production estimates are meeting the LACIE 

project 90-90 goal. These evaluations' are carried out at the 
country level for the U.S.S.R. and at the USGP level for the 
United States. In the United States, early season evaluations 
are based on five-state or seven-state winter wheat data 
projected to the nine-state level and are reported in the 
first and second AA interim reports. Nine-state 90-90 cri- 
terion evaluations are made when the production estimate and - 
standard error become available for the USGP. They are 
reported in the third and fourth interim reports. 

c. Conduct detailed investigations of error sources within 
LACIE production estimates for the USGP, which shall consist 
of the following. 


1 


Relative difference » 


LACIE - SRS 
LACIE 
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First-Order Error Source Investigations. These are 
studies of those errors contributing to LACIE production 
estimates which can be estimated using LACIE estimates, 
reference estimates (USDA/SRS) , and historical and blind 
site data. The effect of each error component is assessed 
by estimating the error in LACIE production estimates 
caused by removing that error. The first-order error 
components to be evaluated are: 

a, . Yield Error Source Estimation 

(1) Yield Bias 

(2) Yield Variance 

b. Area Error Source Estimation 

(1) • Sampling Variance Estimation 

(2) Classification Variance Estimation 

(3) ' Classification Bias Estimation 

(a) Segment Level Bias Estimation 

i. CAMS Wheat Estimation Bias 
ii. Ratioed Wheat Estimation Bias 

1. Small Grains Estimation Error 

2 . Wheat/Small. Grains Ratioing Error 

(b) Countiy Level — Group III Ratio Error 

© Second-Order Error Source Investigations. These investi- 
gations examine in further detail the error components 
identified and/or quantified in the first-order error 
source evaluations. The second-order error source evalua- 
tions are directed toward the investigations of problem 
areas that have been identified during Phase I and Phase’ II 
and toward the examination of error effects that are asso- 
ciated with the operational implementation of new procedures 
and equipment (e.g.. Procedure 1 and the IMAGE 100) in the 
LACIE Phase III analytical process. A detailed aecription 
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of the second-order source investigations is presented in 
section 6.2.5. 

d. Conduct special investigations of LACIE estimates over 

selected areas of the U.S.S.R. and Canada-. These investiga- 
tions include; 

© Evaluation of the CAMS proportion estimates for the 30 
Canadian test sites and 10 ITS's by comparing them with 
ground-observed proportions for the same areas. The 
ground observations are arranged by the Canadian Centre 
for Remote Sensing. . 

The specific ttsks for implementing the above activities are 
identified as tasks 10 through 39 in figure 4.1. Specific 
descriptions of these tasks are provided in section 6.2. 

.4.3 REPORTING ' ’ “ ‘ V ' 

Reporting for Phase III AA consists of the following three types 
of reports. 

a. Special AA management briefings 

b. AA monthly quick- look reports 

c. AA interim and final reports 

4.3.1 SPECIAL AA MANAGEMENT BRIEFINGS 

AA personnel will provide LACIE management with special briefings 
and presentations on the status of AA data acquisitions and ‘ 
special problems during the 1976-77 winter- and spring-wheat 
growing seasons.- These briefings provide timely responses to 
management requests for information about LACIE accuracy through- 
out the growing season. 
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4.3.2 AA MONTHLY QUICK-LOOK REPORT? 

These reports contain an evaluation by AA of the LACIE estimates 
reported in the Crop Assessment Subsystem (CAS) Monthly Reports 
(CMR's), the CAS Unscheduled Reports (CUR's), and the CAS Annual 
Report (CAR) . They are released 1 week following the release of 
a CMR or a CUR if the corresponding SRS or FAS estimates are 
available. Otherwise, they are released 1 week after the release 
of the SRS or FAS estimates. 

4.3.3 AA INTERIM AND FINAL REPORTS 

The interim reports are released in May, August, November, and 
February; and the final report will be released in April. These 
reports describe all the results obtained by AA up to the time 
that each report is written. 

The basic reporting formats and the suggested content of these 
reports are provided in the detailed task descriptions presented 
in section 6.3 of this plan. 
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. . 5. SCHEDULE AND RESOURCE REQUIREMENTS 

The schedule and resource requirements for implementing LACIE 
Phase III AA are presented’ in the following sections. 

5.1 SCHEDULE 

The Phase III AA schedule is presented in figure 5-1. 

5.2 RESOURCE REQUIREMENTS 

The resource requirements for LACIE Phase III AA are shown in 
table 5-1, which summarizes the manpower and computer require- 
ments associated with specific AA tasks or task groups. 
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Figure 5-1.— Phase III AA schedule 



^CEDING PAGE BLANK. WOT FIUA^ 

6. TASK DESCRIPTIONS 

Detailed descriptions of the tasks that comprise AA for Phase III 
are included in the following subsections. 

6.1 PLANNING AND DATA ACQUISITION 

One of the three major elements of Phase ill AA is the planning 
and coordination of analytical and data acquisition activities of 
the program. This involves (!)• the determination of the basic 
data required to support the program, (2) the coordination and 
scheduling of data acquisition activities, and (Z) the monitoring 
of data quality to assure that the data acquired are of satis- 
factory technical quality. 

The AA program 'depends upon the LACIE functional elements [CAMS , 
CAS, Yield Estimation Subsystem (YES), and Data Acquisition, 
Preprocessing, and Transmission Subsystem (DAPTS)] to provide a 
majority of the data necessary for AA evaluations. Specific 
task descriptions are not included in the AA plan for these LACIE 
operations. Only those AA data acquisition tasks where direct 
involvement by the AA.Team or by support personnel is required 
are described in the following paragraphs. 

6.1.1 PLAN DEVELOPMENT 

The initial activities of Phase III AA are directed to the devel- 
opment of an AA plan. This involves definition of the program 
scope, the data/resource requirements, the schedules, and the task 
descriptions of the activities planned for Phase III. The proposed 
activities of the Phase III AA program are documented in this 
Phase III AA plan. 

6.1.2 DATA REQUIREMENTS 

Required inputs from the LACIE functional elements have been iden- 
tified through the planning for AA investigations. The basic 
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inputs are defined in the following subsections. As special prob- 
lems are identified, these requirements may be expanded to satisfy 
newly defined AA needs. 

6. 1.2.1 Classification and Mensuration Subsystem (CAMS) 

CAMS personnel provide the following information to the AA program 
for each 1976-77 analysis performed by CAMS and for which results 
are passed to CAS. This information is also supplied for each 
classification of each ITS. 

a. Segment number. 

b. Day of the year (DOY) , biostage, and data quality indication 
for all acquisitions used by an analyst prior to classification. 

c. Dot labels and designated other/designated unidentifiable (DO/ 
DU) field definitions prior to classification. 

d. DOY, biostage, and data quality indication for any acquisition 
used by an analyst after classification but prior to 
evaluation , 

e. Dot labels and DO/DU field definitions after classification, 
if different from (c) . 

f. Unconditional cluster labels for each pixel as applicable [to 
be supplied via the Information Storage, Retrieval, and Refor- 
matting Subsystem (ISRRS ) ] . 

g. Classifier label applied to each pixel (to be supplied via 
ISRRS) . The bin value of wheat and other categories must be 
kept constant (e.g., wheat = 239, nonwheat = 143). 

h. Prior probabilities used. 

i. Classification run number. 

j. DOY of the acquisitions used in classification. 

k. CAMS evaluation code. 

l. Date of analysis. 
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m. Bias correction applied to classification result. 

n. Identification of dots used to label clusters. 

6. 1.2. 2 Crop Assessment Subsystem (CAS) 

The following data and information inputs are required from CAS 

during Phase III AA: 

a. U.S. and U.S.S.R. CMR's and any related CAS unscheduled reports 
(CUR's), which include LAC IE Phase III estimates of acreage, 
yield, and production as they are developed throughout the 
growing season. 

b. • The standard statistics for area, yield, and production for 

LAC IE Phase III U.S., U.S.S.R., and Canadian estimates, which 
include the standard deviation, the CV, the 90-percent confi- 
dence limits, and the probability of less than 10 percent 
relative error. 

c. A list of segments used in the aggregation, their estimated 
percentage of wheat, percentage of small grains, and acquisi- 
tion date . 

d. Wheat/ small grains ratios used by CAS in aggregations. 

e. Publication of the CAS data base at the end of the crop year 
(e.g., PC, GPC , etc.) 

6. 1.2. 3 Yield Estimation Subsystem (YES) 

The following data and information inputs are required from the 

YES during the Phase III AA: 

a. Adjustable crop calendar (ACC) data used by CAMS to support 
analyst-interpreter classification activity over the blind 
sites, 24 U.S. ITS's, and 10 Canadian ITS's. 

b. Results from the Phase III operational yield model for each 
zone in the USGP, for each zone in the U.S.S.R. where yield 
models exist, and for the years 1964-76 in a stepwise fashion, 
(This data requirement is similar to that for the Phase I and 
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Phase II yield feasibility studies.) These results are 
required to support the first-order yield investigations. 

6 . 1 , 2 . 4 Data Acquisition, Preprocessing, and Transmission 
Subsystem (DAPTS) 

The following data and information inputs are required from the 

DAPTS during Phase III AA. 

a. The following ground-observation data for all U.S. blind 
sites: 

© Aircraft photography (the basis for development of field 
overlays that are used in documenting inventories and 
interpreting signatures; see section 6. 1.4. 2 for specific 
requirements) . 

o Completed fall early season winter wheat and at-harvest 
spring and winter wheat inventories of the blind sites 
conducted by USDA/ASCS personnel using the instructions 
and data recording forms presented in appendix D. 

© Eighteen-day observations of crop height and ground cover 
over 15 wheat fields in each blind site. 

b. The following ground-observation data (to be collected as a 
part of LACIE operations) for all U.S. ITS’s: 

© Aircraft photography (the basis for development of field 
overlays that are used in documenting inventories) . 

o Completed fall wall-to-wall and spring inventories of the' 
ITS’s conducted by USDA/ASCS personnel utilizing the data 
forms presented in appendix E. 

© Completed 18-day periodic observations of approximately 50 
fields from the ITS's throughout the .wheat growing season 
until harvest, to be taken within 3 days of Landsat over- 
flights. (The USDA/ASCS personnel make and record these • 
observations on the forms presented in appendix E.) 
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c. Ground observation of wheat growth-stage data (after each 
growth stage) acquired by USDA/ASCS personnel over U.S. 

ITS ' s . 

d. Copies of mean historical crop calendars based on the last 

15 years, if available, for each acreage stratum for the USGF 
states and Canada and copies of the ACC's, as available every 
2 weeks by CRD. 

e. Agriculture reports;' 

a USDA/SRS reports containing current information on wheat 
acreage, yield, and production for the United States 
at the state level. These shall be made available to AA 
- ■ personnel on the day of release. 

a USD A/FAS reports containing current year information on 
wheat acreage, yield, and production for Canadian and 
U.S.S.R. wheat growing areas. These shall be made avail- 
able to AA personnel on the day of release. 

f. Historical agricultural statistics: 

® USDA/SRS data on wheat acreage, yield, and production in 
the United States for 1970-75. 

a USDA agricultural census data for 1969 and 1974. 

g. Other required data sets as specified by AA personnel to sat- 
isfy special investigations that may be requested by LACXE 
project management. AA personnel will specify any such 
requirements to DARTS as soon as possible after the data 
requirements are identified. 

6.1.3 SITE SELECTION 

For Phase III, the AA team will seject 143 winter wheat and 69 
spring wheat blind sites in the United States and 30 test sites 
in Canada, 
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6. 1.3.1 U.S. Blind Sites 

The U.S. blind sites were selected during the fall of 1976 so 
that aircraft photography could be obtained earlier than was 
the case in Phase II. The location and identity of all U.S. 
blind sites will remain unknown to the CAMS data analysts so 
that these sites can be processed as regular segments. The 143 
•winter wheat blind sites were selected by a random draw strati- 
fied by CRD from five states in the USSGP (Colorado, Kansas, 
Nebraska, Oklahoma, and Texas) and from two mixed wheat states 
(South Dakota and Montana) . The 69 spring wheat blind sites 
were selected from the two mixed wheat states (South Dakota and 
Montana) and from the two spring wheat states (North Dakota and 
Minnesota) „ The distribution of blind sites by state is included 
in appendix B (see table B-3) . 

6. 1.3. 2 Canadian Spring Wheat Test Sites 

Thirty spring wheat test sites will be selected from the Province 
of Saskatchewan, These sites are similar to blind sites except 
that their identity is known to the analysts. 

6.1.4 PREPARATION OF BLIND SITE FIELD OVERLAYS 

Field overlays will be prepared from aircraft photographs of the 
blind sites and will be used to record the land-use information 
obtained by observation on the ground. The following items are 
required to prepare the blind site field overlays. 

6. 1.4.1 Aircraft Maps 

After selection of the blind sites, Landsat imagery is used to 
determine the true position of each site. Analyst-interpreters 
determine these positions using production film converter (PFC) 
products, record the latitude and longitude to the nearest 
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0.1 minute, and plot the position of the segment on a 1:24 000- 
scale or 1:12 500-scale map. These maps are then used by air- 
crews in acquiring the aerial imagery. - 

i: » * i * ' m , / . * 

6 . 1 . 4 . 3 Aircraft Photography 

Aerial photography is collected using color infrared film. If 
possible, this photography is obtained from a high altitude so 
that a single photograph covers the entire site. If this is 
not possible, the flight is made at an altitude of 6000 to 
7200 meters {20 000 to 24 000 feet) and two flight lines are 
flown for each site with a 20-percent sidelap. Four frames are 
collected for each flight line with a 30-percent forward overlap.. 
All imagery must be collected no later than- 4 weeks prior to 
ground-truth collection. Predesignated flight lines are estab- - 
lished for each blind site. . — *• - -■ r . r * 

After aerial imagery is acquired for the blind sites", each frame 
is checked to verify that the site was covered and that the 
imagery is of sufficient quality to be used by. the USDA/ASCS 
personnel in collecting ground-truth data. 

6. 1.4. 3 Field Overlays and Field Segment Kits 

If the imagery is of satisfactory quality to be used by USDA/ASCS 
personnel, transparent overlays are prepared. The overlays are 
then placed in field segment kits that are forwarded to USDA/ASCS 
personnel in the appropriate county for use in acquiring ground- 
truth data. These kits include: 

o A color infrared 2X or 4X print of the segment with boundaries 
on the field overlays . (The 4X enlargement' is used for high- 
altitude photography.) . 

o A topographical map of scale 1:250 000 showing the sample 
segment location and boundaries. 



o Crop identification key with standard annotation for document- 
ing land use. 

© Survey manual with a brief definition of field procedures 
developed at JSC providing guidelines to USDA/ASCS personnel 
for recording ground observations of the LACIE blind sites 
(appendix D) . 

6.1.5 BLIND SITE FIELD DATA ACQUISITION “ 

USDA/ASCS personnel provide complete inventory data based on 
ground observations. The data for each field are annotated on the 
overlay according to the standard crop symbols identified in the 
crop keys provided in the JSC instructions to USDA/ASCS for making 
LACIE segment inventories (appendix D) . These inventory packages 
are to be completed by USDA/ASCS personnel and forwarded to JSC 
to be logged and tracked by DAPTS. _ 

All blind sites in the USSGP that have an early season planted 
inventory will have 15 wheat fields chosen and annotated on the 
overlay by USDA/ASCS personnel: 5 below average stands/ 5 aver- 

age stands, and 5 above average stands. The USDA/ASCS personnel 
will identify the plant height and ground cover of each of these 
fields at this time. Beginning on April 6, the USDA/ASCS per- 
sonnel will begin to revisit these 15 fields in concert with the 
Landsat overpasses so that classification performance can be. 
related to wheat field stands. Also beginning April 6, similar 
observations will commence over all blind sites which have been 
planted. 

As discussed in appendix H, software is being developed which 
will be run in the background mode on the PDP 11/45 and will 
determine, for each of these fields, the amount classified as 
wheat and the amount classified as other for each classification. 
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Prior to the time that this software is ready, manual interpre- 
tation is necessary. To do this, AA personnel will ask CAMS to 
pull the packets; AA will pull the 18-day field observations and 
the ground- truth overlay. The latest imagery used in classifi- 
cation and the classification map from the packet will be pro- 
jected onto the ground truth overlay, and the percentage of each 
of the 15 fields classified as wheat will be estimated and 
recorded. • < - ■■ ■- 

* r . . . j 

Fields 1 through 15, recorded by USDA/ASCS personnel, are assigned 
numbers ranging from 300 through 314 on the ground-truth overlay. 
For those segments with training dots defined on the appropriate 
PFC, each dot or small-field group of four dots will be verified 
in terms of the label given by the analyst versus the ground- 
truth label. Likewise, areas of other crops which are classified 
as wheat are estimated by AA personnel. When the automated system 
becomes operational, these manual interpretations will be used for 
order-of-magnitude verification tests, 

6.1.6 PROCESSING BLIND SITE GROUND-TRUTH DATA TO SUPPORT SEGMENT- 
LEVEL AA INVESTIGATIONS 

The early season blind site ground-observation data will be 
processed according to the procedures used in Phase II. .These 
procedures require the LACIE cartographic technician to plot the 
LAC IE segment boundary on a product 1, 2X photograph. Using the 
area mode feature of the H. Dell Foster digitizer, the technician 
planimeters or measures the segment area in thousandths of a 
square inch on the photograph. Next, the proportions of wheat, 
small grains, abandoned wheat, and abandoned small grains (in 
each case separated into spring and winter) are determined by 
planimetering the area for each of these classes and dividing by 
the total area in the segment. These proportions are used in 
various investigations described in section 6.2. 
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The late season blind site ground-observation data will be proc- 
essed in two ways to determine proportions. The first method is 
designed to obtain quick estimates .of the proportions and the 
second (slower) method is designed to obtain the exact • . 

proportions . 

The first method involves placing a grid containing 400 dots over 
the ground-truth annotated aircraft imagery and determining the 
class of each dot. The proportions in the image are then assumed 
to be the same as tl a proportions in the dot samples . It is 
expected that the, standard error in this procedure will be 
±2.4" percent. Proportions will be determined for wheat and 
small grains, and these classes will be further broken down into 
the categories of spring, winter, harvested, and abandoned. The 
following, list gives the schedule for completing these estimates : 

a. .North Dakota (27) — August 25 

b. South Dakota (19) — August 31 

c. Oklahoma (20 — September 7 

d. Remaining USNGP (69) — September' 19 

e. Colorado (13) — September 21 

f. Remaining USSGP (41), — September 26 

g. Canadian Sites (30) — October 1 

Here the number in parentheses is the number of segments. 

The second method for determining ground-truth proportions 
involves using the Bendix 100 to determine the vertices of each 
field in the segment. These will be stored on a magnetic tape 
along with the identification of the ground-truth class for each 
field. The proportions will then be calculated by the* computer 
routine SPECTL, which is part of the pixel-level processing sys- 
tem described in appendix H. 


6-10 



6.1.7 FIELD DATA ACQUISITION IN INTENSIVE TEST SITES 

The field data acquisition from 24 U.S. and 10 Canadian ITS’s is 
an integral part of LACIE operations. These sites are located 
prior to Phase III operations, and their identities and locations 
are available to all LACIE personnel (see appendix B) . Field 
data acquired from these sites by USDA/ASCS personnel include the 
following: . 

a. Aerial photography (once yearly) 

b. Field maps annotated by USDA/ASCS personnel 

c. Inventories of all fields (Figure E-l in appendix E provides 
an example of the ground-truth data reporting forms . ) : 

e After fall planting for winter wheat areas 

o "At harvest” for spring and winter wheat areas 

d. Periodic 18-day observations of a subsample (approximately 
50 fields) of each ITS coincident with each Landsat overpass 
(Figure E-2 in appendix E gives an example of the ground-truth 
periodic observation form used for recording these 18-day 
periodic observations.) 

These data are forwarded to JSC to be processed, logged by DAPTS, 
and stored in the LACIE Physical Data Library (LPDL) where this 
information is then made available to AA personnel. 

6.1.8 PROCESSING ITS GROUND-TRUTH DATA TO SUPPORT SEGMENT- LEVEL 
INVESTIGATIONS 

Measurement of wheat and small grain proportions (both spring and 
winter) in the ITS will be done by using the Phase II procedure of 
adding field acreages from the inventory list. ' 
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6.2 ANALYSIS AND EVALUATION 

The analysis and evaluation tasks described in this section are 
those presently planned to accomplish the Phase III AA objectives 
described in section 1.1. 

6.2.1 DATA COLLECTION AND PREPARATION 

This task involves collecting the data required by AA from LACIE 
operations and other sources. It is listed as a separate task 
because it involves considerable effort on the part of AA. The 
various data sets required are shown in figure 4-1. 

6.2.2 COMPARISONS OF LACIE ESTIMATES WITH REFERENCE STANDARDS 
AS A METHOD OF ASSESSING THEIR ACCURACY AND RELIABILITY 
DURING EARLY SEASON AND THROUGHOUT THE GROWING SEASON 

A prime concern of the LACIE AA program is to monitor and evalu- 
ate estimates made during early season and at regular intervals 
throughout the growing season. These evaluations are made 
through comparisons with the reference standard (USDA/SRS 
estimates} . 

The statistic used for making these comparisons between the LACIE 
estimates of wheat production, area, and yield and the correspond- 
ing reference estimates is the relative difference (RD) defined 
by: 

_ mCIE - STMDMD x 100% 

LACIE 

where LACIE stands for the LACIE estimate of wheat production, 
area, or yield and STANDARD represents the corresponding refer- 
ence standard estimate. This definition expresses the difference 
between the two estimates as a percentage of the LACIE estimate. 

In the United States these comparisons are made for each state in 
the USGP and for the various regions discussed in section 4.2. 

In the U.S.S.R., they will be made at the country level and 
possibly for certain regions. 
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Significance tests of no difference are made at the region or - 
country level for the LACIE production, area, and yield estimates 
for spring wheat, winter wheat, and total wheat. In order to 
make a significance test, the LACIE estimate (of wheat production, 
area, or yield) is assumed to be approximately normally distrib- 


uted with unknown mean u and variance a 
ihypothesis 


LACIE' 


A test of the 


. . H q : y = STANDARD 

versus the alternative hypothesis 

H a : y / STANDARD 

is then made using this assumption. The test statistic is given 
by 


z = 


LACIE - STANDARD 


'LACIE 


which, under the null hypothesis, is approximately normally dis- 
tributed with mean zero and variance one. The null hypothesis is 
rejected in favor of the alternative at the a-level of signifi- 
cance if 


jzj > z 


a/2 


where z a /2 - g") cr itical point of the standard normal 

distribution. For a = 0.10, z a / 2 = 1*645, and if |z| > 1.645, 
it is concluded that the mean of the LACIE estimator is signifi- 
cantly different from the reference standard estimate. 


These comparisons are designed to detect any abnormal divergences 
between the estimates and reference standards and thus to identify 
for further investigations of potential LACIE problem areas that 
might be associated with the divergences. 
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The results of these evaluations are reported in the Phase III AA 
monthly quick-look reports and in the interim and final AA reports 
which are described in sections 6.3.2 and 6.3.3, respectively. 


6.2.3 DETERMINING IF LAC IE PRODUCTION ESTIMATES MEET THE 90-90 
CRITERION 

A 

Let P be the LACIE estimate of wheat production for the region or 
country, and let P be the true -wheat production of the same region 
or country. The accuracy .goal of the LACIE is a 90-90 at-harvest 
criterion for wheat production, which is given by the following 
probability statement. 

Pr [ i P - P | < 0.1P] > 0.90 (1) ’ 

This states that the accuracy goal is for the LACIE at-harvest 
estimate of wheat production to be within 10 percent of the true 
wheat production with a probability of at least 0.9. 


It is assumed that the LACIE estimate, P, is normally distributed 

. 2 . . 
with mean P + .B and variance a~, where B is the bias given by 

B - E{P) - P 

Under this assumption, equation (1) may be written as ' 


Pr 


-0.1 - 0.9 


B 


P + B 


0.1 - 1.1 


B 


where Z = 


CV(P) 
P - (P + B) 

A 

CP 


< Z < 


P + B 


CV(P) 


> 0.90 


( 2 ) 


follows the staniard normal distribution. 


N (0 , 1) , and CV (P) is the coefficient of variation of P defined by 


CV(P) = 


°P 

E(P) 


a P 


P + B 


(3) 


The term 


B 


P + B 


is called the relative bias of P and is given by 


B 

P + B 


E(P) - P 
E (P) 


(4) 
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It follows that the accuracy goal of LAC IE is attained if 


0 1—11 ® ■ - 


ni- no ® 

* 041 1 * 1 P + B 

- $ 

u.i u,y p + b 

CV{P) 

CV(P) 


(5) 


where $ represents the cumulative standard normal distribution. 

A 

The enclosed region of figure 6-1 indicates combinations of CV(P) 
and relative bias for which equation (5) is satisfied. 


A 

The estimates of CV(P) are provided by CAS and the estimates of 
the relative bias are obtained by estimating B using the method 
described in section 6. 2. 4. 5. If these estimated values fall in 
the enclosed region of figure 6-1, the at-harvest production 
estimate has met the 90-90 criterion. 


Although the "official" evaluation of the 90-90 criterion is 
based on at-harvest estimates, it is of interest to evaluate how 
LACIE is performing throughout the season. When data are avail- 
able for both spring and winter wheat (generally after July) , the 
evaluation is performed in the same way as for the at-harvest 
estimate. In order to gauge how well LACIE is performing early - 
in .the season when only winter wheat data are available, a method 
of projecting the winter wheat results for the 5- or 7-state 
level to the 9-state total harvestable wheat level using Phase II 
results was developed. Since adequate blind site proportions are 
not avaij-able in the early season, the relative difference between 
the LACIE and USDA estimates is taken as an estimate of the rela- 
tive bias. The "projected" relative difference at the 9-state 
level is given by the equation 


RD 

- ” 12 . 


77 


76 


( 6 ) 


where -12.3 is the Phase II final relative difference at the 
9-state level, PD 77 is the current month relative difference in 
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Figure 6-1.- 





1977 for the 5- or 7-state winter wheat production estimate/ and 
RD^g is the corresponding relative difference for the same month 
in 1976. The values for RD ?g are given in table 6-1. 


Similarly, the "projected” CV(P) at the 9-state level is given by 



CV 


77 


'CV 


76 


(7) 


where 5 is the Phase II final CV (P) at the 9-state level, CV^y 
is the current month CV{P) for the 5- or 7-state winter. wheat 
production estimate, and CV ? g is the corresponding CV for the 
same month in 1976. The values for CV ?g are given in table 6-1. 
The 7-state results are used if they are available for both years 


After RDg and CV^ have been calculated, inference as to whether 
the 90/90 criterion has been satisfied can be made by determining 
whether these values fall within the enclosed area in figure 6-1. 
If they do, it is said that the current LACIE estimates support 
(rather than satisfy ) the 90-90 criterion since the determination 
was based on projections which may or may not be accurate. 


6.2.4 ERROR SOURCE EVALUATIONS - FIRST-ORDER ERROR SOURCE 
INVESTIGATIONS 

A major purpose of AA is to attempt to ascertain the nature and 
characteristics of the error in LACIE Phase III production 
estimates. This requires indepth investigations in which the 
error in LACIE production estimates is quantitatively and/or 
qualitatively associated with various causative factors. However 
the error in production depends on its sources in a complex way; 
thus it is unrealistic to assume that the total error component 
can be written as a sum of uncorrelated random components. 
Instead, the effects of the major components are evaluated by 
estimating the reduction in prediction error of production which 
results from eliminating that component of error. These major 
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TABLE 6-1.- PHASE II CV‘S AND RELATIVE DIFFERENCES 


Date 

' ' 

Area, 

state 

Phase II CV 
for production 

Phase II 
relative 
difference 

Feb. 

5 

11 

-4.9 

Mar. 25 

5 

10 

-9.9 

Apr. 8 

5 

8 

-8.5 

May 7 

5 

8 

-1.6 

June 8 

5 

7 

11.4 


7 

8 

1.7 

June 29 

5 

7 

12.7 


7 

7 

4.7 

July 9 

5 

7 

-3.7 


7 

7 

-7.9 

Aug. 11 

5 

7 

-4.2 


7 

7 

-5.6 


9 

6 

-14.7 

Sept. S 

5 

7 

-6.6 


7 

7 

-6.6 


9 

5 

-13.6 

Oct. 8 

5 

7 

-6.6 


7 

7 

-6.5 


9 

5 

-13.8 

Dec. 17 

5 

7 

-7.2 

(final) 

7 

7 

-7.2 


9 

5 • 

-12.3 
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components of production error are called first-order errors. 

They are graphically depicted in figure 6-2. These * irst-order 
errors are the errors contributing to the LACIE production 
estimate which can be quantitatively estimated from LACIE esti- 
mates, reference standard estimates (USDA/SRS) , and historical 
(county census) and blind site data. . Methods for evaluating 
these first-order error components are discussed in sec- 
tions 6. 2. 4.1 through 6. 2. 4. 5. Section 6.2.4. 6 describes the 
method for determining the effect of errors in yield, acreage, 
sampling, and classification on the production estimate. 

6. 2. 4.1 Proportion Estimation Errors 

Proportion estimation errors are .determined for the CAMS estimates 
of small grains and ratioed wheat. ' If CAMS estimates wheat 
directly (i.e., without using ratio) in Phase III, the proportion 
estimation errors for these estimates will also be studied. 


The proportion error for a given blind site segment is X - X, 
where X is the CAMS proportion estimate and X is -the ground-truth 
proportion. These errors are studied by plotting them as a 
function of X for each state in the USGP and for the USGP region. 
Also, statistics for these regions are calculated as follows. 


Let N be the number of segments allocated to a region (state or 
higher level) , and let n be the number of blind sites selected 
randomly from these N segments. For a region, let X^ represent 
the CAMS estimate of the proportion of wheat in the it -h segment 
and let X^ represent the ground-truth proportion of wheat in the 
i th segment, where i = 1,*»*,N. Then the average error is 
given by 


P 


D 


N 



i=l 



( 8 ) 
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Figure 6-2.— LACIE first-order error components. 



















The estimate of u D is given by 


n 



i=l 


where the summation is taken over the n blind sites. 


Letting = X^ 


X^, we may estimate the variance of D by 



Lower and upper confidence limits for the population average ' 
difference are given by 

\ - ° - ! % - 5 + W2 S D (11 > 

where 1 2 _- a /z ^e va l ue of 1 - a/2 percentage point, from the ' 

Student's t distribution with (n - 1} degrees of freedom, corre- 
sponding to the desired confidence level of 1 - a. 


The null hypothesis = 0 (i.e., no bias) is rejected at the - 
a-level of significance if i > t > or equivalently, if 
the confidence interval given by equation (7) does not contain 
zero. 


The quantities D, Sg-, , and y D are tabulated at the state 

L U 

and the USGP levels, and the test is used to determine whether 
or not there is bias at these levels. 
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6. 2. 4. 2 Contribution of the Classification and Ratioing Errors 

to the Ratioed Wheat Proportion Estimation" Errors at~the 
Segment Level 

This section describes the method used to study the contribution 
of the classification and ratioing errors to the ratioed wheat 
proportion errors, which consist of the proportion bias and the 
proportion mean-square error (MSE) . 

Let and x^, i = be the estimates of and x^ 

respectively, for the ith blind site, where is the ground- 
observed ratio of wheat to small grains proportion; is the 
ground-observed small grains proportion; and n is the number of 
blind sites. In this section, r i is the forecast ratio of wheat 
to small grains proportions, and is the CAMS estimate of the 
small grains proportion. , . , ... .. 


The bias (B) and the MSE of the wheat proportion estimate 
obtained after ratioing may be estimated by 


n 


B = 


1 A i 

si rtf* 

i=I ,, ,,, , 


r i x i 


)- 

/ . ti 


( 12 ) 


and 


n 


MBS- £ y /r . x 1 , - *,*.)* ■; 

)n jhuetxS . \ 3* >X" . i 2i'f‘ 1 * 

1" i 


^ .. , 


(ID, 


It is clear that these errors, are both caused by two factors: 

11 r l,« t n t ,l> W» ill >*» XU* Hr » 

the .CAMS .classification., of small .grains and the estimated ratio 

of wheat to small grains.- The contribution of a particular ^ 

error;, .factor, „may„ be,, assessed by. the reduction in the bias, or MSE . 


which, would,, he, achieved if, that error factor were omitted. 
Specif, ically.o, the , .following -formulas -are used in this study , ,, 
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a* Bias estimate with no ratioing error: 

n 

- hE ( r i s i - Vi) (i4) 

i-1 

b. .Bias estimate with no classification- error: 

s " - I E ( J i*i - r i x i) <«> 

i=l 

c. MSE estimate with no ratioing error: 

m = e ' = IE ('A - r i x i ) 2 - < 16 ) 

i=l 

d. MSE estimate with no classification error: 

. mee " = (vi - r i x i ) 2 <i 7 > 

i=l 

These quantities will be calculated at the state and USGP levels 
and a sensitivity analysis will be conducted to measure the 
effect of classify .vtion and ratio error on the bias and MSE 
for ratioed wheat proportion. 

Data required for this investigation are: 

a. CAS ratios for every USGP blind site used in the aggregation 

b. CAMS estimates of spring and winter wheat and either a 
small grains or a spring or winter small grains estimate 
for every 1977 blind site. 

c. Ground-truth proportions for spring wheat, winter wheat, 
small grains, winter small grains and spring small grains 
for every 1977 USGP blind site. 
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6. 2. 4. 3 Classification Bias 


The LAC IE estimate' of wheat acreage for a large area in the United 
States can be written 


n 



i=l 


W i X i 


(18) 


where A is the estimated wheat acreage of the region; is the 
wheat proportion estimate in the i th LAC IE segment, n is the 

number of processed LAC IE segments; and the W. are weights based 

. . 1 1 
on historical and cartographic data. 


A 

Corresponding to A is the true acreage, 


A, which can be written 


n 



i-1 


(19) 


where c^ is the true wheat acreage for the county containing the 
i th segment and W£ is the value of the weight which would give 
perfect Group 'III estimates of wheat acreage for unsampled 
counties. 


We can now write 


X ± = c ± + (x ± - Ci ) + (X t - X ± ) 


“ °i * s i + e i 


( 20 ) . 


where X. is the true wheat proportion of the i th segment, 6.. is 
the sampling error, and is the classification error. Since' 
the sampling was performed in an unbiased manner, we assume 


1 The precise definition of W^ depends on whether the i th segment 
is used as part of a Group III estimate. 
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E(6 i ) - 0? however, we do not assume unbiased classification. 
Instead, let 0 be an average segment bias? i.e., 

= 6 . ( 21 ) 

A /V 

The bias in A is defined by E (A - A) , which is given' by 


B ~ E(A - A) = E 


(I ■*>. - S ■*) 


n 


£=i 

n 


W i E(c i + + E -i) ~ V W* c. 

1 i 11 1 1 

i = l 
n 


= S (W. - W¥)c. + 8 2 W. 
. i=l 1 1 X i=l 1 


( 22 ) 


Note that the first term of equation (22) represents a bias 
caused^by the failure of the Group III ratios to be exact (i.e., 

W i ^ W i* ? whereas second term is the average segment bias 
multiplied by the sum of the w^. 


At present, only the second term of equation (22) is estimated 
since good county— level data are not available for estimating the 
first term. 


The second term is estimated by (1) breaking up the large area 
into strata (not necessarily connected) for which equation (21) 
holds; i.e., the bias is constant, (2) estimating 9 by 

1 n k 

9 = n~ ILi ^ X i ” X i^ • tiie avera 9® proportion error on a segment 
k i=l 

level in the k*7z stratum where n^ is the number of blind sites 
in the kth stratum, and (3) aggregating 8 over the strata. 
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If B represents the estimate of bias for the region resulting 
from classification, a 90-percent confidence interval for B 
(the true bias) can be constructed by 

(B - 1.645a, B + 1.645a) 

A /\ 
where a is an estimate of the standard error of B. 


If we assume Var(e-) = a , (a constant) within the k th stratum, 

1 CK n, g.2 

o (x. “ X. - 0) 

then a , can be estimated by >, — * and Var (B) can 

ck ± ti \ ’ 1 

be estimated by Var(B) = 23 Xf W k ij' w ^ ere is the weight 

for the i th segment in the kt?z stratum. 


6. 2. 4. 4 Estimation of the Within-County Acreage Variances 
Resulting From Classification and Sampling Errors 

In order to estimate the within-county acreage variances result- 
ing from sampling and classification errors, one first obtains 
three basic regression models; namely, true segment proportion 
versus historical county ..proportion, LACIE .^segment proportion 
versus ground- truth segment proportion, and LACIE segment propor- 

tion versus historical county proportion.' Then these regression 

‘ ' " ' 1 ' ‘ y 2 2 ’ 

equations are used to -obtain the estimates for 0 ' + t)„, o'", and 

a a- -4 c where to a-- , and cr^- represent, respectively,, the 
contribution, .resulting from sampling.,, the .contribution resulting 
from., classification,, ..and , the ..variance, „pf, the .residuals, resulting- , 
from- the ; p$gressipn ,of„ the -current ..epunty,, proportion opto,, the , 

,tPUnty, : prppgrti 9 n., As spring ..thatp,, ip mpeh,,, seller, , 

2 7 . . * . 

than a , cr" can be ignored in the calculation • ) Finally, the 

»»• S i ' n *** ** > 1 i'* * ” i,Y “* ' • * *" * 

maximum likelihood estimation technique, assuming normality, is 

•used to obtain the optimal .estimates for sampling -and classifica- 
tion .variances. The detailed description of. this method is. pre- 
sen ted, iin. appendix „C 
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6. 2. 4. 5 Yield Bias and Variance 

TO support 'the evaluation of the 90-90 criterion for production, 
the yield estimates must be tested for bias and the accuracy of 
the corresponding estimated variances determined. In order to 
do this, the bias and variance are tested, as first-order error 
source evaluations, for each zone for all truncations for the 
U.S.S.R. and the USGP using 10 or more years of independent test 
data. This is being done as a test and evaluation task. .The 
data to be used are as. follows. 

a. United States 1965-76 

b. U.S.S.R. 1958-72 

c. Canada 1967-76 

6 .2 . 4 . 6 Production Bias 

The production bias at the state level is given by 


B p = E(P. - P.) 

i 11 

•’ - E(? i> - p i . 

W E(A.Y. ) - A. Y. , (23) 

s 4* X X X 


where P^, A^, and Y^ are the true values of the production, acre- 
age, and yield, respectively, for the i th state in question, and 

/N A 

P^, and are the corresponding estimates for these quanti- 
ties. Assuming A^ and are independent, one obtains 


- E(A.)E(Y.) - A.Y. 

. A A 

If one further assumes that Y. is unbiased, then E(Y.) 

x - 1 



Y i (E(A i ) 


Y.B- 
x A. 


A i 


] 


(24) 

Y . , and 
i 


(25) 
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where B a is the acreage bias for this i th state. The quantities 

i 

;Y. and B are unknown, but an estimate, B p can be obtained by 

•using the^stimates for Y. and B . Thus, 

1 A i 


B Pj. = Y i B A. < 26 > 

The variance of Bp is given by 


Var^B p ^ = Y^ Var^B A j + E 2 ^B A j VarfY^ + Var^B A j Var(Y..) (27) 
and estimated by 


/V / A 

Var ( B t 




^2 

= Y7 Var 
i 


M 


A J A A A / A 

B^ Var (Y . ) - Var/B 


\) 


Var (Y ± ) 


(28) 


For the nine-state level, the production bias estimate B Q is 

Jr 


A 




(29) 


and the estimate of its variance is^Pvar^Bp j. The relative 

bias of the production estimate R(Bp) is estimated by expressing 
the production bias as a percentage of the LACIE-production esti- 
mate; that is, by 


R (B ) 
P 


■ _ r* A 

EVi 


X 100% 


(30) 


6 . 2 . 4 . 7 Effects of Errors in Acreage, Yield, Sampling, and 
Classification on the Production Variance 

The production variance consists of two major error components: 
acreage and yield. The acreage error may be further subdivided 
into sampling and classification errors. The effect of a par- 
ticular error is determined by the reduction in the production 
variance estimate when the error is omitted from the calculation 
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of that estimate. If there is only one yield stratum in a zone 
(state), the production variance is calculated. at the zone level 
and aggregated to higher levels. If a zone contains more than 
one yield stratum, it is subdivided into pseudozones, which are 
the intersections of the zone with the various yield strata. 

The production variance estimate is then calculated at the 
pseudozone level and aggregated to the zone and higher levels. 


Suppose the zone consists of H pseudozones, G ^» G 2 ,••• ,G H , with 
acreage estimates A 21 ,A 22 r * ** '^ZH an< ^ predictions 

Y 2i ,Y z2 , • * • ,Y ZH , respectively. Then the estimate of the produc- 
tion variance at the zone level is given by the following equa- 
tion, which also appears in appendix B of the C7.S Requirement 
Document. 


4 - 2 ( v zi*zi + 4i4i - 

i=l 

t2 ES Y zi x zi (2 E v, 

i=2 1=1 \ jeG i keG Z 




. (31) 


where 


uf . = the estimate of the yield variance for the ith pseudozone 

Z X 



= the area variance estimate for the 1th pseudozone 


= the estimated covariance between Aj in and A^ in G £ 


In order to determine the production variance without a given . 
error term, equation (31) must be rederived with that term 
omitted. Let S ZA , S zy , S zs# an<i S zc t * ie state production 
variances without acreage, yield, sampling, and classification 
errors, respectively. One obtains the following expressions for 
these quantities. 
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where P is 


H 


s L = E ( u li4i - v L u u) 


i=l 


H 


ZY 


= E( 


V 2 Y 2 . - V 2 . tt2 
v Zi Zi v Zi 


i u L) 


i=l 


H i-1 


* * 5 § —ft >5, v) 

E 0 - 5)V Zi*Zi + U Zi A Zi - (1 - 5lV Zi U L] 


i=l 


H i-1 


♦*EEW«-»>2 S v) 

i=2 i=l V k^G £ / 


H 


' = ' (pv 2 .y 2 . + U 2 .A 2 . - pv 2 .u 2 .) 

ZC Z-* V Zi Zi. Zi Zi p Zi Zi/ 


i=l 


H i-1 


* 1 £ § — (’ 5 , > 5 . ’») 


defined on page C-‘ of this report. 


(32) 


(33) 


(34) 


(35) 


2 2 2 2 

Let S rA , S r yi s r g' an & s rC ke the regional-level production 
variance estimates without acreage, yield, sampling, and classi- 
fication errors, respectively. These estimates can be obtained 
from the following expressions. 



( 36 ) 
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R 



Z=1 


R -HR 



Z=1 Z=1 Z'=l 


Z^Z» 


S rZZ' 


ZC 


^ n. 

- E 4 + E E s 


2=1 


rZZ 1 


2=1 Z'=i 
Zy*Z' 


( 37 ) 


(38) 


(39) 


Here R is the total number of zones in the region and S -_, = 0 

xz z 

if Zth and Z'th zones have no yield strata in common. Otherwise, 


s r2Z ' 


V A p V 2 
iL-d rZK rZ'K rK 

K=1 


(40) 


where 

A rZK = the area estiinate £° r the pseudozone corresponding to 
yield stratum K in zone Z of region r 

2 

U rK “ squared prediction error for the K th yield stratum 
common to zones Z and Z ' 

C = the number of yield strata common to the ZfcTi and Z 'th 
zones 


The estimates of the corresponding variances for a country are 
obtained by adding the corresponding estimates for all the 
regions in the country. These computations assume that the 
regional production estimates are uncorrelated. 

6.2.5 SECOND-ORDER ERROR SOURCE INVESTIGATIONS 

Second-order error source investigations are designed to study 
the dependence of the errors in the LACIE system on various 
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causal factors. It is hoped that such studies will suggest 
changes in LACIE procedures that will lead to improved estimates 

The second-order, error source investigations have been grouped 
into two categories: segment level and pixel level. ' Segment 

level investigations start from data relating to a whole segment 
such as a segment wheat proportion. Pixel level investigations 
start from data relating to pixels, or collections of pixels, 
such as dots or clusters. 

6. 2. 5.1 Segment-Level Error Source Evaluations 

The following paragraphs describe the segment-level data error 
source evaluation tasks that are planned during Phase III. 

6. 2. 5. 1.1 Effect of Bias Correction on Wheat Proportions 

The purpose of this task is to determine the error in the LACIE 
Wheat Proportion Estimate (X) before and after bias correction 
has been applied. 


To accomplish this, classification and ground-truth data (X) 
over the blind sites of the DSGP are utilized. Comparisons of 
X versus - X are made as well as X corrected (bias correction 
applied) versus X. Aggregations are made such that the classi- 
fication biases and variances can be examined at the state, sub- 
region, and USGP nine-state levels with or without bias 
correction. 

6. 2. 5. 1.2 Effect of Bias Correction on Small Grains Proportion 
Estimates 

This task is similar to that described in section 6. 2. 5. 1.1 . 
except that small grains estimates are evaluated and no aggrega- 
tions will be performed. 
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6. 2. 5. 1.3 CAMS Evaluation Code 

This task consists of the .invertigation of the relationship of 
the CAMS code (describing the quality of the classification) to 
the proportion error (X - X) observed over the U.S. blind sites 
(211) . Comparison between the proportion error associated with 
the categories of CAMS evaluation codes is made and evaluated 
at state and region levels. 

6. 2. 5. 1.4 Acquisition History 

The purpose of these evaluations is to see if the observed pro- 
portion errors (X - X) depend on the acquisitions (as represented 
by the Robertson biostage) during the winter wheat or spring 
wheat growing seasons . The relationship of error magnitude to 
acquisition pattern is examined. 

The data set for this task includes CAMS classification data and 
ground-truth data during the 1976-77 growing season over the 
USGP blind sites recorded by biowindows. 

6. 2. 5. 1.5 Haze Effects Evaluations 

During Phase III haze effects upon classification are investi- 
gated through the acquisition of optical depth measurements with 
manual radiometers at each ITS at 10-minute intervals from 
30 minutes before to 30 minutes after each Landsat 2 overpass 
of the site. These data shall be collected from February (of 
end of dormancy) to the end of harvest. The solar radiometer 
must be calibrated before and after field use and the data must 
be analyzed with respect to optical depth by November 1, 1977. 

In addition to the haze effects study of the above task, during 
Phase III, three auto tracking solar radiometers are placed in 
the three super sites (Finney County, Kansas; Hand County, South 
Dakota? and Williams County, North Dakota) by the field measure- 
ments team for use during selected overpasses of Landsat 2. 
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Optical depth is used and calculated for six channels at rates 
greater than 1 per second for at least 30 minutes before till 
30 minutes after the Landsat 2 overpass. Wind speed and direc- 
tion, temperature, and relative humidity are also monitored' at 
the same rate over the same period. 

Accuracy evaluations of the manually operated solar radiometer 
involve calculating the mean and standard deviation of the optical 
depth for each of six channels for the 1-hour observation period, 
that is correlated with the Landsat overpass, over the 24 USGP 
ITS 1 s . The correlation between these data and labeling and 
classification omission and commission errors will be determined. 

6. 2. 5. 1.6 Crop Calendar Error Determination 

A major reference utilized by analyst- interpreters in their 
classification procedures is the nominal (mean historical), crop 
calendar and the adjustable crop calendar (ACC) . Since the 
LACIE ACC provides the latest reference, information on the 
stage of development of wheat in an area being classified and 
estimated, it is necessary to determine the accuracy of this 
reference information. This task is -designed to determine the 
accuracy of the ACC- estimates of the wheat growth stage through- 
out the growing season. 

The basic data set for these evaluations is the growth-stage 
data acquired by USDA/ASCS personnel over- the 24 U.S. ITS’s. 

These growth-stage data are acquired utilizing the ground-truth 
periodic observation form presented in appendix- E, figure E-2. 

In addition, growth-stage information is acquired over the 
following Crop Reporting Districts (CRD's): 

a. Texas (IN, IS, 2N, 2S, 5N, 5S, 8N, and 8S)' 

b. Kansas (all CRD's) 
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c. North Dakota {all CRD's) 

d. Montana (1, 2, and 3) 

The CRD growth-stage data are acquired by USDA/ASCS personnel 
using the growth-stage reporting forms presented in appendix F. 

The USDA/ASCS delivers these data to the DARTS at JSC 30 days 
after completion of each applicable growth stage in each CRD 
stratum or equivalent. 

Plots will be made of the ACC outputs (for the ITS's), the mean 
of the ground observations of wheat growth' stages, and the 
nominal crop calendar. Confidence estimates .will be made based 
on the distribution of the ITS ground-truth observations, and it 
will be determined if the ACC results fall -within these limits. 
The relationship of the crop calendar information to known 
episodic events of the current year, such as drought, is inves- 
tigated and reported throughout Phase III AA, along with the 
assessment of the accuracy of the ACC.' 

6.2. 5.2 Pixel-Level Error Source Investigations on Blind Sites 

These investigations will provide a method of more accurately 
assessing the labeling, clustering, and classification perform- 
ance during Phase III. They are based on a new procedure for 
processing the ground-truth data which is described in appen- 
dix H. In this procedure, a "ground- truth tape" is produced in 
which the ground-truth data are presented as an image, analogous 
to the Landsat imagery and to the cluster and classification 
maps produced by CAMS. Each subclass in the ground-truth data 
has its own assigned grey-scale level on the ground-truth tape. 
The subclasses used are shown in table 6-2. In addition, certain 
individual fields (the 15 fields described in section 6.1.5 for 
blind sites and the 50 fields described in section 6.1.7 for' 
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TABLE 6-2.- SUBCLASSES USED IN AA INVESTIGATIONS AND THE CORRE 
SPONDING GREY-SCALE LEVELS ON THE GROUND-TRUTH TAPE. 


Subclass 

Grey-scale 

level 

Fields 1 to 80 

1 to 80 

Alfalfa 

90 

Beans 

' 91 

Corn 

92 

Safflower 

93 . 

Sunflower 

94 

Sudan grass 

95 

Sorghum 

96 

Soybeans 

97 

Sugar beets 

98 

Winter wheat 

99 

Spring wheat 

100 

Barley 

101 

Rye 

102 

Flax 

103 

Cats 

104 

Grass 

105 

Hay 

106 

Pasture 

107 

Trees 

108 

Same as 90-108 except: 


Harvested 

115-133 

Abandoned 

140-158 

Strip fallow 

165-183 

Strip fallow harvested 

190-208 

Strip fallow abandoned 

215-233 

Water 

240 

Homestead 

250 

Idle cropland stubble 

251 

Idle cropland cover crop 

252 

Idle cropland residue 

253 

Idle cropland fallow 

254 



ITS's) also have their own assigned grey-scale levels. The 
image on the ground-truth tape is registered to the correspond- 
ing Landsat image. However, the data on the ground- truth tape 
are at a finer resolution. There are 6 subpixels on the ground- 
truth image for each pixel on the Landsat image. 

In the pixel-level investigations the ground-truth data are com- 
pared with the AI dot data and with the cluster and classifica- 
tion maps produced by CAMS. This allows a determination of the 
actual composition (in terms of ground-truth classes) of each 
dot, of each cluster on the cluster map, and of each class on 
the class map. Computer programs which do this automatically 
are being developed. 

The schedule for the processing of the pixel-level data is shown 
in figure 6-3. It will be seen that the data for Nebraska and 
Texas will not be processed in time to be included in the final 
report . 
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6. 2. 5. 2. I Dot Labeling 

The data set is all the analyst labels for the blind sites, 
starting April 1, 1977, with the small fields bias correction 
sheets and the Procedure 1 starting and bias correction labels 
recorded in the CAM;; packets since June 1, 1977. These data 
will have to be key punched by AA for their analyses. 

In order to investigate dot labeling, the composition of each 
dot is obtained first. This consists of determining the repre- 
sentation of the various ground-truth classes (table 6-2) among 
the 6 subpixels on the ground-truth tape corresponding to each 
dot. The composition of each dot will be printed out since it 
is of interest to rtudy the extent to which the dots represent 
mixtures of different subclasses. Each dot is then labeled with 
the label of the subclass having the largest representation 
among the 6 subpixels corresponding to that dot on the ground- 
truth tape. In what follows, the term "name" will be used to 
indicate the ground-truth label given to a dot in this manner. 
This is to distinguish it from the "label" given to it by the 
analyst-interpreter. It is possible for more than one name to 
be assigned if two or more subclasses have the same representa- 
tion. The names are denoted C™ where C denotes the nth subclas; 

n n 

and the superscript m denotes the number of subpixels of this 

class in the dot. The quantity m/6 is called the "purity" of 

the dot since it is a measure of the extent to which the dot is 

composed of a single subclass. The term "set" will be used to 

indicate the collection of dots named C ra . 

n 

Each dot is also given a class name (as distinguished from its 
subclass name) . The classes are those used to label the dots 
by the analyst. For Procedure 1, they are expected to be the 
following: spring grains (SG) , spring wheat (SW) , winter grains 

(WG) , winter wheat (WW) , grains (G) , wheat (W) , other (0) , and 
a class denoted X which consists of dots that fell on cloud or 
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II 


These 


cloud shadow and, therefore, were "unidentifiable, 
classes will be denoted IK, i = 1, B (i.e., U 1 = SG, 0 ^ - SW, 
etc.). The class names are obtained in the same way as the 
subclass names and are denoted U^, where m is the number of 
subpixels of class IK in the dot. 

Dot labeling accuracy is studied by estimating two confusion 
matrices — one for classes and one for subclasses. The class 
confusion matrix element P (U | u*?) is the probability that a 
dot with the name U 3 ? will be labeled IK by the analyst. The 
elements for which i = j give the probability of correct 
labeling for each set U™. For m = 6, one obtains the probabili- 
ties of correct labeling for pure dots; for m < 6, one obtains 
the corresponding probabilities for mixed dots. The other 
matrix elements give the probabilities for confusion with the 
various other classes. The class confusion matrix shows how 
well the analyst- interpreter is performing at labeling pure and 
mixed pixels of the major classes. 


The subclass confusion matrix element P(U.|c^) is the probability 
that a dot with name will be labeled IK by an analyst. A 
particular subclass C , corresponds to a particular class U., 
and the probability of correct labeling for the dots named C n , 
is given by P(U^, | C™,). The other elements give the probabili- 
ties for confusion with the various other classes. The subclass 
confusion matrix shows how well the analyst- interpreter is per- 
forming on the various subclasses and will indicate any classes 
that pose a particular problem. 

Labeling accuracy will be a function of many factors? a partial 
list of these is given in table 6-3. The effect of these factors 
on labeling accuracy will be evaluated by AA to the extent per- 
mitted by available resources . 
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TABLE 6-3.- FACTORS AFFECTING LABELING AND 


CLASSIFICATION ACCURACY 


Factor 

Description 

1 . 

Crop calendar error 

2 

Acquisition history (including biostage) 
Used by analyst 
Used in. classification 

3... 

Percentage of wheat in segment 

4 ' ' . 

Percentage of small grains in segment 

5 

Percentage of other crops and idle cropland in 
segment . 

6 

Percentage of pasture in segment 

7 

Percentage of grasses in segment 

8 

Percentage of irrigation in segment 

9 

Soil type 

10 

. Elevation . 

11 

Crop moisture index 

12 

Palmer index 

13 ' 

CAMS evaluation code 

14 

Analyst 

15 

Items 3 to 8, historical 

16 

Field size 

17 

Amount of grazing 

18 

CAMS PCC a for type 1 dots 

19 

CAMS PCC for type 2 dots 


a The CAMS PCC refers to the agreement between the analyst 
dot labels and the assignment of class by the classifier. 
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Finally, a study will be made to determine whether the probability 
of a dot being correctly labeled is higher if the analyst label 
agrees with the classifier label for that dot. 

6. 2. 5. 2. 2 Clustering 

Three aspects of clustering will be studied:' Cluster composition 
and' purity, cluster labeling accuracy, and the cluster confusion 
matrix. The data set to be used is all blind site cluster maps 
from April 1, 1977, to the end of Phase III. These data cur- 
rently are in the form of DTRM tapes which AA is receiving from 
ISSRS. 

Cluster composition is the percent of the subpixels in a given 
cluster that belong to each of the major classes. This is 
determined by comparing the cluster map with the image on the 
ground-truth tape. The major classes .are SG, WG, G, 0,-and Y, 
where Y consists of both designated other (DO) and designated 
unidentifiable (DU) areas. The set of all these classes will be 
denoted V... The "purity" of a cluster is the percentage of the 
total number of subpixels in the cluster that belong to the class 
with the largest representation. The composition and purity of 
clusters are of interest since they indicate how well the clus- 
tering algorithm is able to separate the classes into relatively 
"pure" clusters. These quantities will be studied as a function 
of segment, region, and acquisition history. 

Cluster labeling will be studied by first naming each cluster 
with the name of the class V.. having the largest representation 
of subpixels in the cluster. The cluster is correctly labeled 
if the label given by the labeling logic corresponds to this 
name. In the case of nearest-neighbor labeling logic, an i 

incorrect label may result from analyst- interpreter mislabeling 
of the dot used to label the cluster or poor performance by 
the labeling logic. If the identity of the dots, which were 
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used to label each cluster, can be determined, these two sources 
of error will be studied separately. Cluster labeling accuracy 
will be studied as a function of cluster purity, acquisition 
history, segment, and region. 

Two confusion matrices are estimated for clusters — a class con- 
fusion matrix and a subclass confusion matrix. The clustering 
confusion' matrices will be studied as a function of segment, 
region, and acquisition history. 

The class confusion matrix element PfVjlV^) is the probability 
that a subpixel of class will appear in a cluster labeled 
Vji For i = j, this gives the probability of correct clustering; 
for i ? j , it gives the probabilities of the clustering algorithm 
assigning a subpixel of name to the various confusion classes. 

The subclass confusion matrix is defined in an analagous fashion. 

The matrix element P(Vj|C n ) is the probability that a subpixel 

of subclass C is assigned to a cluster labeled V.. If C is a 
n 3 n 

subclass of V j , the assignment is correct; otherwise, it is not. 
The study of the P (V ^ 1 C^) allows one to determine if any particu- 
lar subclass or group of subclasses is not getting properly 
clustered. 

6. 2. 5. 2. 3 Classification Accuracy 

Classification will be studied by estimating the classification 
confusion matrices for classes and subclasses. The data set to 
be used is the classification files as the DTRM tapes described 
in the previous section.- For classification, the classes are . 

SG, WG, G, 0, X, and T, where T indicates pixels which have been 
thresholded by the classifier. They are denoted W ^ , j = l, 6.. 

The subclasses are the same as for dot labeling and clustering. 
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The class confusion’ matrix element P(W..|w^) is the probability 
that a subpixel of class W. is classified as class W. . For 
i =» j, this gives the probability of correct classification; 
for i ^ j , it gives the probabilities of misclassif ication into 
the various other classes. 

The subclass confusion matrix element P(Wj(C n ) is the probability 
-that a subpixel of class C n is classified as K_j. If C n is a sub- 
class of W j , the classification is correct; otherwise, it is not. 
The study of PCW^jC^) allows one to determine if any particular 
subclass or group of subclasses is not getting properly classified. 

To the extent permitted by available resources, the effect of 
the various factors in table 6-3. on the classification confusion 
matrix elements will be evaluated. 

6. 2. 5. 2. 4 Effect of Crop Height and Ground Cover on 
Classification f t -curacy 

This study will use the crop height and ground-cover data 
acquired every 18 days for 15 selected wheat fields in each 
blind site. The computer program SPECTL (appendix H) will com- 
pute the probability of correct classification for each of these 
fields and this will be plotted as a function of crop height, 
Robertson biostage, ground cover, and "green number." Means and 
other relevant statistics will be calculated at the segment, 
state, and regional levels. 

6. 2.5. 3 Pixel-Level Investigations for ITS 1 s 

CAMS will process the 24 ITS's in the United States in the same 
manner as regular LACIE segments are processed. The objective in 
this analysis is to determine labeling and classification accu- 
racies. For winter wheat, this will be done for all three . 
procedures used to determine winter wheat proportions, namely 
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the Phase II procedure, the small fields procedure, and Proce- 
dure 1. For spring wheat, only Procedure l will be investigated 
because it is the only procedure to be used with spring wheat. 

The ground-truth data consist of two complete inventories of the 
test site, one in the fall of 1976 and the other in the spring 
of 1977. Between these periods, updates are provided every 
18 days, coincident with the satellite overpass, for a subset of 
the fields in the test site. This subset is called the 18-day 
fields. The performance evaluation analysts will use these 
updates to correct the ground-truth designations for the 18-day 
fields on the ground-truth map and keep them current. Also,, on 
the basis of the information available for the 18-day fields, 
the analysts will interpret on a current PFC image to update the 
designations for all other fields in the test site; this will 
provide a complete, wall-to-wall ground-truth image for the test 
site. A similar photointerpretation will be performed to obtain 
ground-truth designations for the fields outside the test site 
but within the LAC IE segment containing the test site. 

In the Phase II and the small fields procedures, labeling and 
classification accuracy will be determined for each classifica- 
tion for the test site, and a set of 30 fields chosen at random 
from all the fields in the test site. 

For Procedure 1, labeling accuracy and proportion error will be 
determined for each classification, both for the area within the 
test site and for the whole LACIE segment. 

The "true" proportion of the area within the test site will be 
determined by adding the acreages of the fields in the site. For 
the whole segment, the "true" proportion will be determined by 
applying a bias correction, based on ground-truth labels, to the 
machine proportions of the segment. 
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The results of these investigations will be reported in the 
interim reports and at least one ITS story will be presented in 
the quick-look following the CAS CMR. Thus, .the ITS can fulfill 
a role not possible by the blind sites — illustration of accu- . 
racy using annotated imagery. 

This effort starts February 1977 when the first inventories are • 
received and ends in October when the harvest inventory is 
received. 


6. 2. 5. 4 Second-Order Investigations on the IMAGE 100 System 

These investigations are designed to evaluate results obtained 
with the IMAGE 100 Hybrid Procedure 1 operational system. 

6. 2. 5. 4.1 Study of Proportion Error 

The data set will consist of 24 ITS's in the United States and 
10 ITS's in Canada. The IMAGE 100 proportions will be determined 
for the test sites (which are smaller than the whole segment) and 
compared with the corresponding ground-truth proportions deter- 
mined in the manner described in section 6. 2. 5. 3. The method 
used to make the comparison is described in section 6. 2. 4.1. 

6.2. 5. 4. 2 Effects of AI, Acquisition History, and Bias 
Correction on Proportion Estimation Error 

The IMAGE 100 processor and data from eight U.S. blind sites will 
be used in an experiment wherein each site will be analyzed by ■ 
three AI 1 s to give a Procedure 1 "raw" and a "bias-corrected" 
estimate of the proportion of small grains in each segment. The 
segments will be of two types? ncunely, those having acquisitions 
in all four biophases and those having only early season acqui- 
sitions. The segments will be chosen at random from the blind 
sites for which detailed ground-truth data are available. 
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The objectives or the experiment are: (1) to evaluate the perform- 

ance of Procedure 1 in terms of absolute proportion estimation 
error and its repeatibility over AI’s, (2) to make comparisons 
between "bias-corrected" and "raw" Procedure 1 estimates, and 
(3) to determine if the performance is better when acquisitions 
from all biostages are used than when only the early season 
acquisition is used. 

Analysis of variance will be used to test for the effects of 
AI’s, time (i.e., early season versus all acquisitions), method 
(raw versus bias correction estimates) , and their interactions. 

6.2.6 ACCURACY ASSESSMENT EVALUATION OF THE NEW SAMPLE STRATEGY 

During LAC IE Phase III a new sample strategy is being tested 
over Kansas and North Dakota. AA evaluations are designed to 
examine how well the new sampling strategy is doing over these 
f reas through comparisons between the old and new strategies . 

'ine o\rluations utilize (a) blind site LACIE classification and 
round-truth data collected during the 1976-1977 growing season 
over Kansas and North Dakota, (b) 1976-1977 LACIE and USDA/SRS 
production, area, and yield estimates over these same states, 
and (c) 1974 USDA/SRS historical county data also for Kansas and 
North Dakota. 

The AA evaluations of the new sample strategy consist of : - 

a. Subtask 1 making comparisons (relative differences and CV's) 
between the 1977 LACIE estimates of production, area, and 
yield over Kansas and North Dakota from the old and new 
sample strategies, and estimates made by USDA/SRS. Fig- 
ure 6-4 further defines these comparisons. 

b. Subtask 2 determining and comparing aggregated acreage bias 
for old and new sample strategies over Kansas (winter wheat) 
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Figure 6-4.— AA new sample strategy (NSS) evaluations. 
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and North. Dakota (spring wheat) « Figure 3-4 also provides 
an additional description of these evaluations. 

c. Subtask 3 performing regression analysis using 1977 blind 
site classification data, ground-truth data, and USDA/SRS 
1974 historical census data to obtain three basic regressions. 

© True segment proportion vs. historical stratum proportion 

o LAC IE segment proportion vs. ground truth segment 
proportion 

e LAC IE segment proportion vs. historical stratum proportion 

Using the results of these regressions, a sensitivity analysis is 
performed in which the acreage error is quantified (a) with both 
the sampling and the classification errors, (b) with the sampling 
error and no classification error, and (c) with the classification 
error and no sampling error. Then the reduction in error from- 
(a) -(b) and (a) -(c) is examined to establish which, if either, 
is contributing more error to the acreage estimate. 

6.3 AA REPORTING 

There are three types of AA reports: Special management brief- 

ings, monthly quick-look reports, and interim and final reports. 

6.3.1 AA SPECIAL MANAGEMENT BRIEFINGS 

AS required to support LACIE reporting functions, special brief- 
ings are prepared for LACIE project management relative to the 
current status of LACIE operational data development, particu- ■ 
larly with respect to special problems that could affect the 
accuracy of final LACIE at-harvest estimates of wheat acreage, 
yield, and production in the USGP, 

In addition, AA. personnel support LACIE project management 
requirements to brief NASA, USDA, and NOAA upper-level manage- 
ment on the status of LACIE outputs and the progress being made 
toward satisfying the 90-90 criterion. 
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6.3.2 AA MONTHLY QUICK-LOOK REPORTS 

The quick- look reports contain an evaluation by AA of the LACIE 
estimates reported in the CMR's and the CAR. The quick- look 
reports are released one week following the release of a CMR or 
a CAR. The CMR's and CAR's contain the official LACIE estimates 
of wheat production, area, and yield, and the corresponding 
statistics. The true wheat production, area, and yield for the 
particular region or country are, of course, unknown. Therefore, 
to ascertain the accuracy of the LACIE estimates, comparisons 
are made with a reference standard. In the United States, the 
reference standard consists of the most recent (at the time of 
the comparison) estimates released .by the USDA/SRS. In foreign 
countries, the reference consists of the most recent estimates 
released by the USDA/FAS. 

In addition, the quick-look reports will contain (a) significance 
tests of no difference (between the LACIE estimates and the ref- 
erence standard) at the region or country level; (b) results of 
blind site analyses of proportion estimation error; (c) classi- 
fication bias aggregated to the regional level; and (d) within 
stratum acreage variance due to classification and' sampling 
errors. 

6.3.3 AA INTERIM AND FINAL REPORTS 

The interim reports are released at regular intervals throughout 
the crop season. They contain the results of the previous quick- 
look reports, and the results of all other AA investigations up 
to the time each report is written. 

Each interim report is built up from the previous one by includ- 
ing data that became available' during- the interim period. Tech- 
nical comments on each report are solicited from a variety of 
sources and are used to upgrade subsequent reports. Early and 
mid-season evaluations are made in the first and second interim 
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reports; late season and at-harvest evaluations are made in the 
third and fourth interim reports. 

The fourth, interim report also serves as a draft for the final 
report, which contains material similar to the interim reports 
but covers the entire year. 

The planned schedule for release of the LACIE interim AA reports 
and the final Phase III AA report is as follows: 

a. First interim report - May 15, 1977 

b. . Second interim report - August 15, 1977 

c. Third interim report - November 15, 1977 

d. Final interim report - March 1, 1978 

e. Final Phase III AA report - June 1, 1978 

The planned contents for the four interim reports and the final 
report are shown in figure 6-5. The AA interim reports require 
approval by the AA manager and the Chief of the RTEB of NASA/ JSC 
prior to their release for LACIE project review and evaluation. 

The final LACIE Phase III. AA report requires review and approval 
of the following persons prior to its release for distribution: 

a. D. E. Pitts, Manager, LACIE AA, NASA/JSC 

b. J. D. Erickson, Chief, RTEB, NASA/ JSC 

c. J. L. Dragg, Chief, Applications System Verification Branch, 
NASA/JSC 

d. F. G. Hall, LACIE Project Scientist, NASA/JSC 

e. J. Hill, Assistant Deputy Project Manager for LACIE, NOAA 

f. J. Murphy, Assistant Deputy Project Manager for LACIE, USDA 
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Figure 6-5.- Summary of Phase III AA interim and final reports. 








g. R. B. Erb, Chief, LACIE Project Office, NASA/JSC 

h. R. B. MacDonald, Chief, Earth Observations Division, NASA/JSC 
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APPENDIX B 


LAC IE PHASE III BLIND SITES, TEST SITES, 
AND INTENSIVE TEST SITES 


The LACIE ITS and blind-site selections are given in the follow- 
ing tables . 


Table 

Sites 

B-l 

LACIE Phase III U.S. ITS's 

B-2 

LACIE Phase III Canadian ITS's 

B-3 

Phase III Test Site and Blind-Site 


Distribution 


B— 1 



TABLE B-l.- LACIE PHASE III U.S. INTENSIVE TEST SITES 



As me ic a ted by ground- truth data: 
WW * winter wheat 


SW » spring wheat 

51 KW ■» spring and winter wheat 


ORIGINAL PAGE 13 
OK POOR QUALITY 
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ORIGINATE PA'GS IS 
OR POOR QUALITY 

TABLE B-2 LACIE PRASE III CANADIAN INTENSIVE TEST SITES 


[ $e?nent 


provlnco 

Saskatchewan 
British Colunbi-i 
Saskatcho 'in 
Saskatchewan 
Albert* 

Manitoba 

Manitoba 

Alberta 

Alberta 

Manitoba 


Hclfort 

Dawaeft Crock 

bellale 

Swiit (Torrent 

Lethbridge tRayraondJ 

Stony Mountain 

Starbuck 

Olds 

Ft. Saskatchewan 
Alton* 


“•As indicated by ground-truth data: SW * Spring vhoat 


[~ Center coordlr.atos/sne s 

:e 

Wheat 

I LAt . . N 

Long. , W 

S. n * 

Kn 

typo* 

cn 

• 

rv 

ui 

104*44 * 

2*10 

3.2-16 

sw 

S5-4S* 

120*11' 

2 ‘ 1 0 

3.2*16 

sw 

51° 55* 

107*28’ 

2*10 

3.2*16 

SW 

50*19* 

107*53* 

2*10 


sw 

49*30* 

112*48* 

3*10 

3.2*16 

sw 

50*04 * 

97*21' 

2*10 

3.2*16 

sw 

49*47* 

97 *29 * 

2*10 

3.2*16 

sw 

Si *54’ 

in* «• 

2*10 

3.2*16 

sw 

53*38* 

113*07* 

2*10 

3.2-1& 

sw 

49*12 * 

97*35* 

1*5 

1.6-8 

sw 


UK1G1HAL S AGE B 
. )it ' POOR QUAUTll 
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TABLE B“3 PHASE III- TEST SITE AND 
BLIND-SITE DISTRIBUTION 


State 

Number of sites 

Colorado 

15 

Kansas 

31 

Nebraska 

26 

Oklahoma 

20 

Texas 

18 

Minnesota 

20 

Montana 

29 

N. Dakota 

30 

S. Dakota 

23 

Total 

212 

Canadian Test 
Sites 

30 
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APPENDIX C 


CONTRIBUTION OF SAMPLING AND CLASSIFICATION TO ACREAGE 

ESTIMATION ERROR 

This appendix describes the calculation of the contribution of 
sampling and classification errors to the variance of the LACIE 
production estimate. 


C.l APPROACH 

The variance of the LACIE acreage estimate for a large area 
(e.g., 2one) can be written 


V 


2 


E 

i 


vJ 


[ 1 ) 


2 

where is the variance of the acreage estimate for the ith 
county and is a weight which .depends on the size of the 
county, the number of segments in the county, etc. (Refer to 
the CAS Requirements Document, appendix B for details.) 

2 

The variance represents a mean-squared deviation between the 
LACIE estimate for the county wheat proportion and the true 
county wheat proportion. This variance is caused mainly by 
two factors:’ sampling errors and classification errors. 


In accuracy assessment, it is desirable to quantify the contribu- 
tion of each of these error sources to the large area production 
estimate. The LACIE production estimate depends on acreage and 
yield estimation errors in a complicated way; hence, it is 
unrealistic to assume the error in the production estimate can 
be written as a sum of uncorrelated random variables representing 
acreage and yield errors. Instead, the effect of a particular 
error source is measured by the reduction in the LACIE production 
variance which would be achieved if that source were eliminated. 


C-l 



It will be assumed that the i th county acreage error variance 

2 . 2 2 2 2 2 
a . can be written o. = a + A a , where a is a contribution- due 

-L C S * C * 

2 2 

to classification, and X a g is a contribution due to sampling.. 
To determine the effect of no classification error, the variance 
of the LACIE production estimate will be made using pa? instead 


of where 


p is an estimate of the ratio 


,2 2 

X a s 


"c + xZ °r 


Similarly, 


the effect of no sampling error is estimated by replacing of by 
2 i 

(1 - The following two sections describe the methods 

employed for estimating sampling and classification variances 

and the function p. 


C . 2 ACREAGE REGRESSION MODELS 

For counties with one sample segment, the LACIE estimate of the 
i th county wheat proportion can be- written 


y L * c i + (*i - <V + (y-j. ~ x i) 

= C. + e. + <5. - (2) 

i “*• 

where 

y i - LACIE estimate of the wheat proportion in the sampled segment 
= true (current year) proportion of wheat in the county 
x^ ~ true proportion of wheat in the sampled segment 
e i = sampling error = - C^ 

5^ = classification error = y^ - 
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It will be assumed that for some reasonably large area (e.g., a 
zone) the errors and <5^ have the following properties: 


e i and 5^ are uncorrelated • (3) 

E(e i ) = 0 (4) 

E(6i('Xi) *» A*x^ + Q (5) 

v{E i ) = °s (6) 

= o* (7) 


It is also assumed that there is a linear model relating the 
current year counties proportions, c.^, to the historical propor- 
tions which will be denoted by 2^,* i.e., 

^ « o + {Zj + ^ (8) 

where E(C-) = 0, V(£.} - cr^ r Cov(?.',e,) = Cov(?.,<5.) = 0 and 
i x n xx xx 

a and 8 are regression coefficients. 

From the above assumptions and definitions, three basic 
regression models are obtained: 

a. True segment proportion vs. historical county proportion — ■ 
from the definition of 


x i - c i + £ i 


It follows that 


' a + + g + s i 


E{x i ) = a + 8Z i 

V (x . ) s vl + a 2 
r H s 


(9) 


(10) 


LACIE segment proportion vs. ground truth segment proportion 

<5. 
i 

y i = x i + 6 x* 


from the definition of 6. 

l 
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It follows that 


E (y^j x i> ' = + X*x^ + 0 

V(y i |x i ) = a 2 

Writing X = 1 + X*, one obtains 

F (y^ IXf) = Xx i + 0 (11) 

V(y i ix i ) = a 2 ■ ' (12) 

c. LACIE segment proportion vs. historical county proportion — 
from equations (9) through (12) , 

E(y ± ) = E x [e ( y ± ! 3C ± )] = E x (X XjL + 0) = X(a + BZ i ) + 9 (13) 

' v(y i> ~ B xT v< i r ii K i , 3 + * 4 * **{4 + 4 ) 


(14) 


As stated previously, one would like to estimate p. = 


,2 2 

X a s 


„2 a- 1 2 2 

a c + \ °s 


None of the three regression models permits an estimate of 

2 2 2 2 

separately from o H ? i.e., one can only estimate a s +■ a fJf not 

alone. If current year county proportions were available, 
could be estimated, but since this is not the case, 

X 2 (o 2 s + o£) 

= — ^ TJ~1 2T ke estimated instead of p. If 


°c + 


.2 / 2 ^ 2 \ 

x 1°S + a n) 


2 

<< cr s (a reasonable assumption) then p* » p. 
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C. 3 NORMALITY ASSUMPTIONS — MAXIMUM LIKELIHOOD ESTIMATION OF p* 

Suppose a given zone has m blind site segments 'and n ordinary (i.e., 
not blind site) segments, and let the blind site segments be num- 
bered 1 to m. It is assumed that ground truth wheat proportions 

are available for the blind sites and LACIE estimates 

are available for all the segments. It is also assumed 

that historical wheat proportions are available for the 

counties containing the segments. If a 2 << a 2 , so that p ^ p* 

s 

the regression models equations {9 through 14) can be used 
to obtain 

E(x^) = a + 3Z^; V(x^) ^ a 2 i =? l,2,***,m 

E(y il K i* = Ax i + 9; v (y i J = a c i = 1, 2, • * ♦ ,m 
E(y.) = 6 + Xa + XgZ.j V (y . ) = X 2 a 2 + c 2 i = m+1 , • • • ,m+n 

If there is one segment per county, then .the errors and 5^ 
are independent for different values of i, and hence the likeli- ■ 
hood function of the sample can be written ■ ■ 

m m+n 

l = n f(x i ,y i ) n h(y i ) (15) 

i=l i=m+l 

where is the joint density of x i and for i = l,m and 

h(y^) is the density of for i = m+l,m+n. 

m m 

The function n ^ x i ,y i^ can wr itten jQf f{x^,y_^) = 

i-1 i-1 

m 

g (x^) where fCy^Jx^) is the conditional density 

i=l 

of y i given x i and g(x i ) is the density of x^ 
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If normality is assumed, n f(x.,y.) *= TT — - — 

i=l 1 1 £* a c / 2 ? 


— ^2“ ^i “ ^ x i “ I / — exp 

2o Z | o / 2 tt i 

C 1=1 ) S 


m+n i 1 , 

1 h<yil “ “T W^Sf. iSa ,yi ' 


- e - x & z i >' 


Letting Q “ -21ogL - log2ir, 

D T T 

■> . 7. ■. 2 . , 2 2 . . m , m . n 

Q = m log a + m log a + n log.(a + a a ) + *"T + — 2 + “5 2~ 2 

s ' s a a a + A o 

esc s 


where 

m 2 

D m * l (y i - Xx i - 81 

m 2 

f (x. - a - * Z i> 


m+n 

T n= E (y i“ 

i=m+l 


Act - 0 - A0Z i > 


One attempts to maximize L by finding a stationary point of Q: 


2 (x. - a - f$2 

1 3Q _ 1 1 

2 3a „2 


m-rn 

E x <Yi - - 6 - x 6 z i> 

*-n f 1 


2 , >22 
a + A a 
c s 
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1 9Q 

2 3F ~ 


m 

£ z i (*4 - a 
1 1 


m+n 

- 02.) S XZ i (y i “ Xa ~ 6 “ Xe V 

1 m+1 


2 , ,22 
a + A a 
c s 


m 


1 19 

2 36 


£ (y, - Ax. 
1 1 - 1 


m+n 

-9) E (y i ' Xa ■ 0 
, m+1 


« z i> 


n ■) ■) 
a * + A a 
c s 


= 0 


0 

‘ ( 18 ) 


(19) 


m 


„ Z X l ty i " Xx i 

1 3Q 1 x 1 - 1 

2 c)X ~ „2 


- m+n 

0) - nXa s + ' £ (BZ i + a) (y i “ 

, i=nt+ 1 

~2~2 


Aa - 9 - 


cl + A a‘ 
c s 




2 2 
A aZT 

+ 7 2 .2 2\2 0 

. ( c c + X a sj 


( 20 ) 


3Q_ 

9a' 


m 

“2 

a c 


n 


A 2 a 2 + a 2 
s c 


n 


'a 2 c 2 + a 2 ) 2 

L S C 1 


= 0 


( 21 ) 


3Q_ 

3<E 


m 


n\‘ 


2 

X °s 


V 


(• 


2 , .2 2 \2 
+ A cr 
c s 


= 0 


( 22 ) 


Equations (17) through (22) must be solved for the parameters 
a, 8, 0, A, o 2 , and a 2 . If a, 8, G, A, a 2 / and a 2 represent the 
solution to equations (17) through (22) , then the invariance 
theorem for maximum likelihood estimation can be used to 
obtain 


P = 


C2-2 
X q s 

o 2 + a 2 5 2 
c s 


( 23 ) 


as the maximum likelihood estimate of p. 
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The equations (17)' through (22) are nonlinear but can be 

solved using numerical techniques.. Newton's Method was used to 

solve the equations for this report; i.e., if u* k * is an estimate 

of the solution vector u <= (a, P, §, X, S 2 , a 2 ) at the ktA step, 

c s 

then . . 


U (k + u = u (k) . F -l f ( u (kl) ' (24) 

where f(u^) = (f ,f fi ) ^ is the vector of the left sides of - 


equations (17) through (22) evaluated at u 
“ 3u, * 


(k) 


and F * 


In practice, it was slightly simpler to use the parameter 
transformations . . 



s 

r 2 1 

(25) 




and 

s = X 2 cr 2 + Q 2 
s c 

. (26) 


and solve for a, g, 8, X, r, and s. Again, the invariance 
theorem can be used to give 

p = X 2 r (27) 

C. 4 ACCURACY OF p 

Since p is an extremely complicated function of the data, it is- 
impossible to write down the variance of- p for finite sample 
sizes m and n. However, the asymptotic variance of p can be 
estimated using the information matrix; i.e., if 


T 



(28) 


08 



and g(u) = g ^cx, 3 , 0 ,£ ,o^ ,5^ j is a differentiable function of the . 
parameters u, then the variance of g{u) is asymptotic to 


where 


[g* (u) ] T V 1 g' (u) 

g *(u) == /-ia, . . . 

9 • l?V au e/ 


(29) 


Thus, in our case, g(u) = 2 ~ 

a a„ + o 
s . c 


g‘ (u) = 


0,0,0, - 2Xo<(x 2 o| + °l )~ 2 + o 2 ) 2 '(a 2 +\ 2 a 2 ) 


.2 2 

X °s 


(°i ♦ > 2 °ir 


(30) 


To estimate t, the observations {x^}, {y^}, and {.Z^} and the 

, / ~ ~ ~ ~ *2 - ~2 \ . 

estimated parameters (a,3,8,A,0 , and a 1 were substituted into 

2 

the matrix’ H = (h^j) = . Then equation (29) was used 

l 3 

to obtain an approximate variance for p. 
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APPENDIX V 

instructions to usda/ascs 
FOR 

t A.CXE BLIND SITE 

Sgment inventory 

XJVCIE BT,IN° SITE 
15-FIELD PERIODIC 
OBSERVATIONS 


© 





NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 

LYNDON D. JOHNSON SPACE CENTER 
Houston. Texas 77058 


REPLY TO 

ATTN OF; TF4-76-11-31 


I!QV & 2' 19/6 


0 VJTlO. v 

& fp* % 

9 v* 


TO: County Executive Director 

FROM: Bobby £. Spiers 

Production Specialist 

SUBJECT: Transmittal of Material for LACIE Segment Inventory 


Enclosed is the material to be used to complete the identification of 
crops in the LACIE segment located in your county. The information 
requested is essential to the evaluation of the effectiveness of our 
experiment and we ask that you make the inventory and return the data 
to us within two weeks. - 


The instructions explain in detail what needs to be done. Read tho 
.instructions and if you have questions, let us know. Thank you for 
your assistance. 


&J4y. f X, 

Bobby E. Spiers 




uW- 


Enclosures 

1. Instructions 

2. Post Card 

3. Return Label 

4. Color Infrared Print(s) 


d*c2w 


D-l 


'-v.-vanvT 



FALL 

INSTRUCTIONS 

FOR 

LAC IE SEGMENT INVENTORY 
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TABLE OF CONTENTS 


I. Introduction 

II. Data Collection Procedures 

III. JSC Contact Point 

IV. Due Dates and Hailing Procedures 
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I. Introduction 
A. Background 

The LACIE (Large Area Crop Inventory Experiment) is, an inter-, 

agency experiment in the use of Landsat (formerly called Earth Resources 

Technology Satellite! and meteorological data to identify and inventory 
» 

crop production. Participating agencies include the Department of 
Agriculture, the National Aeronautics and Space Administration, and 
the National Oceanic and Atmospheric Administration. Within the 
Department of Agriculture, participating agencies are the Agricultural 
Stabilization and Conservation Service, Economic Research Service, 

Foreign Agricultural Service, Agricultural Research Service, Soil Con- 
servation Service, and Statistical Reporting Service. The overall 
general objectives of the LACIE are to determine utility and cost 
effectiveness of satellite and surface derived data sources to monitor 
large area crop (v/heat) production and assess the impact of agricultural 
and meteorological conditions on production estimates. The utility of 
the information produced will be evaluated on the basis of its objec- 
tivity, timeliness', accuracy, and its expected value for policy and 
program decision making. 

LACIE reports are based on data extracted from 5x6 mile segments 
that have been randomly placed throughout the wheat producing region of 
the United States, In order to determine our accuracy, it is necessary 
that we know what is actually in our sample segment. The information 
requested for the segment that has been identified and forwarded to you . 
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Is essential for a successful evaluation of the project. The enclosed 
color. prints have been obtained only over the selected site in your 
county to support ground data collection,. This is the only copy of the 
print and additional copies are not available for distribution to offices. 

B. Authority 

The USDA LACIE Project Manager has requested that the Agricultural 
Stabilization and Conservation Service provide this function and they 
have accepted the assignment. You should have already gotten an authori- 
zation from your State office concerning this task. If you have not, you 
•should contact them at once. 

C. Requirements of the ASCS County Office 

You are being asked to do the following: 

1. Review the set of Instructions. 

2. Visit the segment location and identify the land uses, 
even if the segment falls outside your county. 

3. Check over your work and return the completed inventory 
as soon as possible. 

II. Data Collection Procedures 

A. Supplies 

1. Color infrared print or prints, 

2. Mylar overlay. 

3. Topographic map with segment location. 

4. Standard crop key. 

5. Crop stage development key. 

6. Forms to record plant height, ground cover observations 
and evaluation comments. 

f 
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7. Return post card and r ai.n mailing tube label, 

B, In some cases, all of the segment will not be covered by the 
photo. Complete the survey for that' portion outlined on the photo. 

C, Procedures 

1. You are. required to identify all fields within the segment 
boundaries using codes as indicated on the attached crop 
key (see attached LACIE segment classification), 

2. Use black or red ball point pen for all coding directly on 
the mylar. 

3. The photos are provided as a base for field pattern and 
references. 

a. All field identification should be based on actual 
ground conditions on the day that you visit the segment. 

b. If there are any differences between the photo and the 
ground, then footnote each field that is different and - 
explain on evaluation form, 

c. Fields that are currently idle crop land should be 
marked as I/(code) and specifically identify the type 
of cover, i.e., ST * stubble; CC = winter seeded or 
volunteer growing crop; RE = disked or harrowed with 
residue; F = fallow (clean tilled). 

Use the evaluation form for all comments on any unusual 
crop condition or practice (irregular, replanting, drought, 
etc.) . 
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5, If there are any crops in the segment for which there is no 
code t select an unused symbol- and indicate its meaning on 
the evaluation -form, 

6. Assess the average wheat crop stages while completing the • 
segment inventory and enter it on the evaluation form upon 
completion. 

III. JSC Contact 

A. If there are any problems, contact the person listed below, 

B. Review procedures and crop key before going into the field and 
. ' contact the Johnson Space Center if there are any questions. 

Bobby E. Spiers 

U.S. Department of Agri culture/ ASCS 
NASA -.Johnson Space Center 
Large Area Crop Inventory Experiment 
Houston, TX 77058 

Phone: Commercial - A/C 713-483-4623 

FTS - -525-4623 

IV. Due Date and Mailing Procedures 

A. Upon receipt of data from the Johnson Space Center, complete the 
enclosed card and return it to JSC. 

B. Field information should be collected within 10 days after 
receipt of material by your office, if at all possible. 

1, Upon completion of field survey, fill out enclosed forms 
and return with photos. 

2. Return all evaluation forms and photos in a mailing tube, 
using the provided return label. 

C. Thank you for your cooperation and effort in assisting LACIf 
in this vital area of the experiment. 
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SAMPLE SEGMENT 


With Photo and Mylar 
(Not to Scale) 



S ample : U/A These were winter wheat fields hut are currently 
being plowed under due to- the effects of drought. 
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STANDARD CROP KEY 


KEY 

W 

sw 

G 

H 

A 

P 

C 

SF 

$U 

SG 

SR 

SY 

SB 

FX 

T v 

R 

B 

X 

BN 

0 

(Crop)/H 

(Crop)/A 

I/CCode) 


CROP TYPE 
Winter Wheat 
Spring Wheat 

Grass (not cut for hay and no fence) 

Hay (any visible signs of hay activities) 

Alfalfa 

Pasture 

Corn 

Safflower 

Sunfl ower 

Sudan Grass 

Sorghum 

Soybeans' 

Sugar Beets 
Flax 

Trees 

Rye . 

Bari ey 

Homestead - ncnag, lakes, ponds, etc. 

Beans 

Oats 

Crop has been harvested, 1977 crop 

Crop has been abandoned; footnote and explain, 1977 crop 

Idle crop land/ ST = stubble; cc = growing cover crop; 

RE = residue; F = fallow (clean tilled) 


1. Do not use the code W or SW for any crop other than wheat. 

2. If there are crops in segment for which there is no code, select an 
unused symbol and Indicate its use on the evaluation fora. 


3. Use standard key for all identification. 


4. Use ball point pen for all coding on mylar. 
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CROP STAGE KEY 


1 STAGE KEY 

STAGE 

DESCRIPTION 

1.0 ‘ 

PI anted 

Seed was put in the ground. 

2.0. ’ 

Emerged 

• When one leaf per plant is visible. 

3,0 

Jointed 

Defined as when the first node of the stem 
is visible. 

4.0 

Heading 

Defined as the stage when the base of the 
rachis (or head) reached the same height 
as the ligule (or base of the shot leaf). 

5.0 

Soft Dough. 

At this stage the crop is starting to turn 


color. The kernels can be easily deformed 
. when pressed between the fingers, but no 

• "milk" or liquid should exude under such 
pressure. 

‘6.0 Hard Dough The kernals readily part From the head. 

The grain is firm and though it may be 
■dented by pressure of the thumbnail, it is not 
. easily crushed. The characteristic color of 

the grain has become more distinct. The 
leaves are brown, dry, and shrunken. Wheat in 
this stage may be swathed in some areas. 

7.0 Harvested or . Straw is brittle and dull yellow at this 

Harvestable stage. The grain (if nof. harvested yet) • 

is hard and breaks into fragments when 
crushed. 
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EVALUATION FORM 


Segment No.: County: State: 

# 

Name: ' ’ Date: 


Man-Hours Required to Complete Survey: Mileage: 

l. . Based on your assessment of the development of wheat in the segment 
while completing the survey, what is the average wheat stage for the 
segment? See attached Crop Stage Key. Is the crop development this 
year in the segment normal, ahead, or behind as compared to previous 
years? Explain. Enter Crop Stage: _ 


U. Comments, footnotes, and additional crop key used: 


III. Comments on the effects qf drought and/or winterkill: 


LV. Comments and recommendations for improving these procedures for future 
surveys: 
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Plant Height Segment Number ; 

and Observation Date 

Percent Ground Cover Observations » 

* ' * 

1) During and/or after your segment inventory complete this form and 'obtain plant heigi 

and cover in 15 segment wheat fields. . , 

2) These fields should represent a range from below average to above average stands of 

Wheat for’ the sample segment only. ■ _ j 

3) Select five fields in each category below average , average , and above average based 
on your judgment of the overall average condition of wheat in the sample segment-only, 

: 4) Plant height and percent ground cover should be measured at the same location within 

the field and should represent the average condition of the field. 

5) Numbers should be assigned to fields according to the table below, fields 1-5 below 

average, 6-10 average, 11-15 above average. Fields can be selected and data collected 
in any order. • . « 

6) Plant height should be measured to the nearest inch. 

; 7) Ground cover should be measured according to the following codes: 1(0-1955), 

j 2(fC-39%), 3(40-5955), 4(60-79%), 5(80-100%). This measurement is taken by looking 
: straight down at the ground and estimating the percent of soil that is covered by the 

.. crop, and % assigning the appropriate code to the field. 

1 Daua Collection Requirements : • 

-You are required to select fields and determine which of three categories the field 

• should be assigned to. Select the next available field number in that category. Enter 

• the field from the most accessible corner or turn row. Proceed into the field at least 

; '20 paces to a spot that you feel represents the average condition of the field and 

; proceed to obtain plant height and percent ground cover. Write the field number, on 

• the overlay at the exact location that the measurements we re obtained and circle the 
■ ' number. Continue with the inventory or field selection. 


Below Average 

Average 

Above Average 

gcjTjnn 

■ 

Percent 
Ground 
Cover Code 

Field 

No. 

sn 

Percent 
Ground 
Cover Code 

Field 

No. 


Percent 
Ground 
Cove'” Code 

l 



6 



11 



2 



7 



12 



3 



8 



13 



4 



9 



14 



5 



10 



15 
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APPENDIX E 

FORMS USED IN TAKING INVENTORIES 
OF LACIE INTENSIVE TEST SITES 



APPENDIX E 


■ FORKS USED IN TAKING INVENTORIES 
OF LACIE INTENSIVE TEST SITES 

' * 

The following forms are used to record ‘ground observations in the 
LACIE ITS’ at 

Figure Form 

E-l Sample Ground Truth Inventory Form 

E-2 Sample Ground Truth Periodic Observation 

Form 

E -3 Sample form for Ground Truth Data Collection 

System Rainfall Measurements 

E-4 Sample Yield Form 
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Figure E-i.— Sample Ground Truth Inventory Form. 
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Figure E- 
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TEST SITE f 30 (SALINE, KANSAS! 


FIELD f DATE OBSERVEO 
MONTH DAY 


OATE RAINED 


MONTH DAY 


. GROUND TRUTH DATA COLLECTION SYSTEM , 

rainfall measurements’ 

RAINFALL ' FIELD* DAT^ OBSERVEO 



• • RAINFALL ' 
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Figure E-4.— Sample Yield Form. .. 







APPENDIX F 


CHOP GROWTH STAGE REPORTING FORMS 



GROWTH STAGE DATES FOR 10* DEVELOPMENT .. 
MONTH AND DAY OF MONTH .. 


Check one: 

□ Spring Wheat 

□ Winter Wheat 
Crop Yea r 


ITS SEG. NO. 

CRO if i 

PLANTING 

DATE 1 

EMERGENCE 
1 DATE 1 

JOINTING 

DATE 3 

HEADING 

DATE* 

SOFT 

DOUGH 

DATE* 

RIPE 

DATE' 

HARVEST 

DATE 1 

1 

j 

j 

| 

1 

I 

j 

1 

! 

j 


• 



• 




** Date at which 103 of fields in CRD were planted or emerged* respectively. 

1 Dote at which 103 of fields In CRD had - begun to Joint or hcad ( respectively. . 

* Date at which 102 of fields in CRD hod benun to enter soft dough stage (turning color to greenish-yellow to yellow). 

" Date at which 102 of fields In CRD are ripe (hard dough) .stage or when they were swathed, (Indicated swathed if 

applicable). 

* Dote at which 102 of fields in CRD have been harvested either as standing grain or out of swath. 



GROWTH STAGE OATES FOR 50* DEVELOPMENT 
MONTH AND DAT OF MONTH 


Check one*. 

D Spring Wheat 
n Winter Wheat 
Crop Year 


ITS SEG. NO. 

CRD n 

PLANTING 

DATE 1 

EMERGENCE 

DATE 1 

JOINTING 

DATE* 

HEADING 
DATE* ' 

SOFT 

DOUGH 

DATE* 

'RIPE 

DATE 4 * 

. HARVEST 
DATE* 


1 

' 

1 

! 

. 

• 

• 


- 


- 


n Date at which 50« of fields In CRD were planted or emerged, respectively. 

1 Date at which 50% of fields in CRD had -begun to Joint or head, respectively. 

’ Date at which 50% of fields in CRD had begun to enter soft dough stage (turning color to greenish-yellow to ysllow). 

% Date a* which 50% of fields In CRD are ripe (hard dough) -stage or when they were swathed, (Indicated swathed if, 

applicable). 

. * Date at which 50t- of fields In CRD have been harvested either as standing grain or out of swath. 



Check one: 

0 Spring Wheat 

GROWTH STAGE OATES FOR 90% DEVELOPMENT □ Winter Wheat 



1 Date at which 90 % of fields in CRD were planted or emerged, respectively. 

* Date at which 902 of fields in CRD had begun to Joint or head, respectively. 

* Date at which 90% of fields in CRD had begun to enter soft dough stage (turning color 'to greenish-yellow to yellow). 

' Date at which 90% of fields in CRD are ripe (hard dough) stage or when they were swathed. (Indicated swathed If 

applicable). 

3 Date at which 902 of fields in CRD have been harvested either as standing grain or out of swath. 
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APPENDIX H 

PHASE III AA PIXEL-LEVEL 
DATA PROCESSING SYSTEM 


In Phase III, AA will conduct pixel-level investigations .for the 
first time. These are designed to monitor the performance of 
the various components of the LACIE system (analysts, clustering 
algorithm, and classification algorithm) and to investigate the 
sources of error in these elements. These investigations are 
described in section 6, 2, 5, 2. This appendix describes the data 
processing system that has been designed to support the pixel- 
level investigations. In addition, this system automatically 
produces some data (such as segment wheat proportions) that are 
used In segment-level investigations. 

The system involves the processing of ground-truth data, CAMS 
classification maps, CAMS cluster maps, and CAMS analyst- 
interpreter-selected dot labels. A flow diagram of this AA data 
processing system is presented in figure H-l. The Phase III AA 
processing described in this figure includes four major steps. 

1. CAMS DTRM tape preprocessing 

2. AA ground- truth preprocessing 

3. CAMS analyst dot label preprocessing 

4. AA classification/label accuracy processing 

The first three steps are necessary to prepare for the fourth 
one. This last processing step is done using the computer pro- 
grams 3PATL and SPECTL on the PDP 11/45. 

i 

Since only about 10 percent of the segments on the DTRM tapes are 
U.S. blind sites, a preprocessing program is required. This 


H-l 



program will strip the blind site data from the DTRM tape and put 
it onto another tape containing only blind site data. 

The DTRM tape contains all of the classification maps pnd condi- 
tional and unconditional cluster maps for all segments processed 
by CAMS/ beginning with the segments processed using the small 
fields procedures. Since the identity of the blind sites is to 
remain unknown to the CAMS analysts, AA will provide blind site 
identification cards as input for the DTRM tape-searching program. 
When data for a blind site are located on the DTRM tape, they will 
be copied onto the AA blind site class/cluster map tape (the 
first step in the processing task) . 

The second step is the conversion of ground-truth maps into 
ground-truth tapes suitable for subsequent processing. This 
initially involves the use of the Dell Foster equipment which 
produces field vertices cards. These cards are converted to a 
tape by the program BTAPE to produce a tape with a format similar- 
to the tape created by the Bendix 100 Interactive Drafting System, 
.This format contains the x and y coordinates of the vertices of 
the agricultural fields from the aerial photographs of the blind 
sites. The resultant product is a nine-track computer-compatible 
tape (CCT) containing a field identification and field 
coordinates. 

The Dell Foster equipment will be used for early season process- 
ing until the Bendix 100 software package for the Interactive 
Drafting System becomes available and is checked out. The sched- 
ule for the beginning of ground-truth map processing using the 
Bendix 100 is June 15, 1977. The Bendix 100 tapes will be proc- 
essed by the programs PHA3E1 and PHASE2 to produce a Universal 
format ground-truth tape containing AA crop labels at the sub- 
pixel level, registered to the Landsat coordinates. This AA. 
ground-truth tape will be coded as per the AA codes presented in 



table 6-2. The ground-truth tape is the product of the second 
step in the AA processing system. 

The third step requires SF3 to acquire the SF4 CAMS pixel-label 
sheets which contain the analyst dot labels and classification 
dot labels prepared during the Phase III operational processing. 
These sheets were initially used in operations when the small 
fields procedures were implemented and will be used throughout 
the remainder of Phase III. From these sheets, SF3 will keypunch 
dot label cards to be used as input to the AA accuracy calcula- 
tions (step 4) . The AI dot label card deck or tape is the out- 
put of step 3. As Procedure 1 implementation progresses, the 
CAMS/CAS interface tape may be used to obtain the analyst- 
interpreter dot labels instead of the analyst-interpreter dot 
label cards. 

The AA classification/labeling processing element utilizes pro- 
grams SPATL and SPECTL to compare the class/cluster map tape, 
the ground- truth tape, and the analyst-interpreter dot labels. 

This comparison is done at the pixel and segment levels, which 
generate data for subsequent AA evaluation. For additional 
information regarding this processing step, refer to technical 
memorandum LEC-10620 (ref, 4), entitled "Requirements Document 
For Phase Til LACIE Accuracy Assessment." 

Later in Phase III, an AA data file will be used as indicated 
in figure H-l. The PDP 11/45 will be used to store the three 
basic inputs on the disk (class/cluster maps, ground truth, and 
analyst dot labels) . The fourth-element programs will use 
this disk as a data base. 

Figure H-l indicates the processing responsibilities of SF3 and 
SF12. The responsibilities of SF3 are contained within the large 
dashed lines, with the remaining elements being the responsibility 
of SF12. 
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In order to assure that the results from the programs SPECTL and- 
SPATL are correct; some of the data should be processed- inter- 
actively on the IMAGE 100 system. It is anticipated that about 
one segment per week or about 5 percent of the blind sites should 
be processed in this manner. About 4 hours of IMAGE 100 time 
per segment will be required. 
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Figure H-l.- 


Phase III Accuracy Assessment data 


processing system flow diagram. 
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APPENDIX I 


ABBREVIATIONS, ACRONYMS, AND SYMBOLS 



AA 

AA-Q1 

ACC 

Agromet 

AI 

Biowindow 

biophase 


Biostage 


APPENDIX I. 

'ABBREVIATIONS, ACRONYMS, AND SYMBOLS 
Accuracy Assessment. 

LACIE Phase XI AA Report for February 1976. 

adjustable crop calendar. . 

.. agricultural/meteorological . 

■ analyst- interpreter. - ■ 

or biological window — a Landsat data acquisi- 

tion period that is related to the biostages 
..of wheat development. The LACIE approach is 
based on the judgment that wheat can be sepa- 
rated .adequately from other crops by analysis 
'of up to four acquisitions of Landsat data 
during the growing season; The biowindow may 
be .updated if there is a significant lag or 
advancement in the current crop calendar. 

The sequence chosen includes acquisitions 
during, the following biowindows: 

1. Crop establishment — from 50 percent 
’ tillering to 50 percent jointing (bio- 
stage 2.3 to 3.0). ... .. ; 

.2. Green — from. 50' percent jointing to 

50 percent heading (biostage 3.1 to 4.0). 

3, Heading — from 50 percent heading to 
- ...50. percent soft dough {biostage 4.1 to 
.... o*0). 

V4’. Mature — from 50 percent soft dough to 

50 percent harvest (biostage 5.1 to 6.0). 

. biological stage — the specific stage of 
development of a crop which can be recognized 
by a major change in plant structure; i.e., 
emergence after germination, jointing, heading, 
soft dough, ripening, and harvest, which are 
represented by integers on the Robertson 
Biometeorological Time Scale. 
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Blind site 


BMTS 
CAMS 
CAR *’ 
CAS . 
CCEA 


Classification 


Classification 

error 

CMR 

COM 

CRD 


Crop calendar 


a LAC IE sample segment that is a part of the 
LAC1E operational random sample set of seg- 
. ments used, in the LACIE aggregations. The 
blind' sites are not identified to analysts 
*/in order to ensure normal processing of the 
blind site segments. However, in Canada, 

■ where only the blind site segments are being 
..processed, the analyst obviously knows their 

location. '' * 

- Biometeorological Tima Scale. 

'Classification and Mensuration Subsystem. a 

CAS 'Annual Report 1 

.Crop Assessment Subsystem. 

Center for Climatic and Environmental 

■ Assessment — an organization of the National 

Oceanic and Atmospheric Administration, 
Columbia, Missouri. . 

in computer-aided analysis of remotely sensed 
data, the process of assigning data points to 
specified -classes by a testing process in 
which the spectral properties of each unknown 
, data point are compared with spectral proper- 
ties typical of -the subject being classified. 

a measure of the degree to which the LACIE 
CAMS overestimates or underestimates the wheat 
acreage in one or more LACIE samples. 

. CAS Monthly Report. 

classification on microfilm. 

Crop Reporting District — a geographical area 
used by the U.S. Department of Agriculture 
for the collection and reporting of agricul- 
tural information; each district consists of 
several counties. 

a calendar depicting the biostages of the 
major crop types within a specified region 
during a calendar year. 
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Crop calendar 
adjustment 

CUR 

CV 

DAPTS 

DO/DU 

DOY 

First-order error 
source evaluations 

Group 2 segment 

Group 3 segment 

IE 

' IMR 
ISRRS • 

ITS 

JSC 

LACIE 


an adjustment made to the normal crop calen- 
dar on the basis of current meteorological 
data. 

CAS Unscheduled Report. 

coefficient of variation (standard deviation 
divided by the mean) . 

Data Acquisition, Preprocessing, and Trans- 
mission Subsystem. 

designated other/designated unidentifiable, 
day of the year 

evaluations to identify the major components 
of error in LACIE Phase III production esti- 
mates; these components can be statistically 
estimated with techniques described in 
section 6.2.4 of the report. 

LACIE segment in a county that historically 
produces small quantities of wheat/smali 
grains; samples are allocated with probability 
proportional to size. 

LACIE segment in a county that historically 
produces very small quantities of wheat/small 
grains; estimates are based on the changes in 
acreage of group 1 and 2 segments from year 
to year . 

Information Evaluation. 

IE Monthly Report. 

Information, Storage, Retrieval, and Reformat- 
ting Subsystem 

intensive test site - a LACIE segment in the 
United States or Canada over which detailed 
crop information is collected by using 
ground and airborne observation equipment; 
these ITS segments are a separate set of 
segments from the operation LACIE segments 
used in the aggregations. 

Lyndon B. Johnson Space Center of NASA, 
Houston, Texas. 

Large Area Crop Inventory Experiment. 
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Land sat 


LEC 

LPDL 

LPP 

MSS 

Multitemporal 

analysis 

NASA 

NOAA 

90-90 criterion 

PCC 

PFC 

PPS 

HD 

RTEB 

Sample segment 
Sampling error 


Land Satellite — formerly called ERTS (Earth 
Resources Technology Satellite); operates in 
a circular, Sun-synchronous, near-polar orbit, 
of the Earth at an altitude of approximately 
915 kilometers; orbits the Earth, about 14 times 
a day and views the same scene approximately 
every 18 days. 

Lockheed Electronics Company, Inc. 

LACIE Physical Data Library. 

LACIE Performance Predictor. 

Multispectral scanner System or multispectral 
scanner — the remote sensing instrument on 
Landsat that measures reflected sunlight on 
various spectral bands or wavelengths. 

analysis of data sets over the same area 
acquired at different times during the grow- 
ing season. 

National Aeronautics and Space Administration. 

National Oceanic and Atmospheric Administration. 

criterion that the LACIE U.S. Great Plains 
production estimate is 90 percent accurate, 
at harvest, 90 percent of the time (in com- 
parison with the true value) . 

probability of correct classification. 

production film converter. 

probability proportional to size. 

relative difference. 

Research, Test, and Evaluation Branch. 

a 5- by 6-nautical-mile area selected by 
stratified random sampling; information is 
recorded by the MSS and transformed into 
computer-compatible tapes and film products. 

a measure of the degree to which the esti- 
mated wheat acreage in the LACIE sample seg- 
ments does not represent the wheat acreage 
contained in the survey region being sampled. 
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Second-order error 
source evaluations 


USDA 

USDA/ASCS 

USDA/FAS 

USDA/SRS 

U.S. Great Plains 


WMO 

YES 


evaluations to further qualitatively break 
down the identified first-order components 
into factors that can be related to causal 
elements in LACIE methods and procedures, 

U.S. Department of Agriculture. 

USDA Agricultural Stabilization and Conserva- 
tion Service. 

USDA Foreign Agricultural Service. 

USDA Statistical Reporting Service. 

an area encompassing the nine states of 
Colorado, Kansas, Minnesota, Montana, Nebraska, 
North and South Dakota, Oklahoma, and Texas; 
it is divided geographically into (1) the U.S. 
southern Great Plains, which includes Colorado, 
Kansas, Nebraska, Oklahoma, and Texas, and 
(2) the U.S. northern Great Plains, which 
includes Minnesota, Montana, and North and 
South Dakota, 

World Meteorological Organization. 

Yield Estimation Subsystem. 


SYMBOLS : 

A 

P wheat/small-grain proportion estimate. 


P 


W 


proportion of wheat harvested. 


o standard deviation . 

CV (W) CV for production estimate. 


Pq T proportion of wheat/small grains based on 

identification of each field in the blind 
site or ITS by USDA/ASCS personnel. 


1-5 



Q 




APPENDIX J 
REFERENCES 



APPENDIX J 


REFERENCES 


1, Large Area Crop Inventory Experiment (LACIE) Crop Assessment 
Subsystem (CAS) Requirements. Vol. IV, LACIE-C00200 (rev. B) , 
JSC-11329, July 1976. 

2, Large Area Crop Inventory Experiment (LACIE) Yield Estimation 

Subsystem (YES) Requirements. Vol. Ill, LACIE-C00200 (rev. A), 
JSC-11340, Sept. 1976. ' 

3, Large Area Crop Inventory Experiment Crop Assessment Sub- 
system Software Requirements Document. JSC-10009, Nov. 25, 
1975. 


J-l 


*U S. GOVEfiNMENf PR'NriNG OFFICE; 1978-771 083/709 







